孙立文的博客

Ubuntu下用Vim编辑且不改变GBK编码的文件

Liwen — Wed, 14 Mar 2012 10:53:18 +0000

Ubuntu下用Vim编辑且不改变GBK编码的文件

sudo vi /var/lib/locales/supported.d/zh

添加以下内容：

zh_CN.GB18030 GB18030
zh_CN.GBK GBk
zh_CN.GB2312 GB2312

执行

sudo locale-gen

vi ~/.vimrc

添加以下内容

set encoding=utf-8
set fileencodings=cp936,gb18030,ucs-bom,utf-8,big5,euc-jp,euc-kr,latin1
set termencoding=utf-8
language zh_CN.GB18030

以上配置可以实现打开编辑GBK编码的文件，但保存时并不改变编码。

Tags: Linux, VIM
没有评论

笔记：Ubuntu下快速开始使用Python Thrift

Liwen — Wed, 08 Feb 2012 12:56:28 +0000

本文介绍如何在Ubuntu 10.04下安装Apache Thrift并用Python写一个Demo。

apt-get install libboost-dev libevent-dev python-dev automake pkg-config libtool flex bison sun-java6-jdk
wget http://www.apache.org/dist//thrift/0.8.0/thrift-0.8.0.tar.gz
tar zxvf thrift-0.8.0.tar.gz
cd thrift-0.8.0
./configure
make
sudo make install
sudo pip install thrift

编辑接口文件 hellowworld.thrift:

service HelloWorld {
    string ping(),
    string say(1:string msg)
}

编辑 server.py

#!/usr/bin/env python
 
import socket
import sys
sys.path.append('./gen-py')
 
from helloworld import HelloWorld
from helloworld.ttypes import *
 
from thrift.transport import TSocket
from thrift.transport import TTransport
from thrift.protocol import TBinaryProtocol
from thrift.server import TServer
 
class HelloWorldHandler:
  def ping(self):
    return "pong"
 
  def say(self, msg):
    ret = "Received: " + msg
    print ret
    return ret
 
handler = HelloWorldHandler()
processor = HelloWorld.Processor(handler)
transport = TSocket.TServerSocket("localhost", 9090)
tfactory = TTransport.TBufferedTransportFactory()
pfactory = TBinaryProtocol.TBinaryProtocolFactory()
 
server = TServer.TSimpleServer(processor, transport, tfactory, pfactory)
 
print "Starting thrift server in python..."
server.serve()
print "done!"

编辑 client.py

#!/usr/bin/env python
 
import sys
sys.path.append('./gen-py')
 
from helloworld import HelloWorld
 
from thrift import Thrift
from thrift.transport import TSocket
from thrift.transport import TTransport
from thrift.protocol import TBinaryProtocol
 
try:
  transport = TSocket.TSocket('localhost', 9090)
  transport = TTransport.TBufferedTransport(transport)
  protocol = TBinaryProtocol.TBinaryProtocol(transport)
  client = HelloWorld.Client(protocol)
  transport.open()
 
  print "client - ping"
  print "server - " + client.ping()
 
  print "client - say"
  msg = client.say("Hello!")
  print "server - " + msg
 
  transport.close()
 
except Thrift.TException, ex:
  print "%s" % (ex.message)

thrift --gen py helloworld.thrift
python server.py
python client.py

参考: Thrift the missing guide

Tags: Apache Thrift, Python
2个评论

ab(Apache Benchmark)运行apr_socket_recv报错

Liwen — Wed, 11 Jan 2012 18:34:57 +0000

在Mac OS X Lion下运行ab会报如下错误：

apr_socket_recv: Connection reset by peer (54)

比如：

$ ab -n 1000 -c 10 http://localhost:8888/
This is ApacheBench, Version 2.3 <$Revision: 655654 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/
 
Benchmarking localhost (be patient)
Send request failed!
Send request failed!
Send request failed!
apr_socket_recv: Connection reset by peer (54)

新版的Apache已经解决了这个问题，可以获取源代码并编译如下：

wget http://mirrors.kahuki.com/apache//httpd/httpd-2.3.16-beta.tar.bz2
tar jxvf httpd-2.3.16-beta.tar.bz2 
cd httpd-2.3.16
./configure

我只需要一个ab，所以只编译support目录即可。

cd support
make
#...

重新运行ab

./ab -n 1000 -c 10 http://localhost:8888/

Bingo! :)

对于老一些的版本的Apache，需要打补丁

wget https://www.rtfm.ro/download/patches/ab.patch --no-check-certificate
patch -p0 < ./ab.patch

会有一行出错，不要理，继续往下走。

patching file support/ab.c
Hunk #1 FAILED at 670.
Hunk #2 FAILED at 1683.
Hunk #3 FAILED at 1767.
3 out of 3 hunks FAILED -- saving rejects to file support/ab.c.rej

然后按前面的步骤编译Apache的support目录。

参考：这里和补丁作者。

Tags: AB, Apache, Mac
没有评论

Ubuntu下PHP哈希冲突漏洞快修

Liwen — Thu, 05 Jan 2012 01:21:09 +0000

方法如下：

sudo apt-get install php5-suhosin
sudo sed -i "s/;suhosin\.post\.max_vars/suhosin\.post\.max_vars/" /etc/php5/fpm/conf.d/suhosin.ini
grep suhosin.post.max_vars /etc/php5/fpm/conf.d/suhosin.ini
sudo /etc/init.d/php5-fpm restart

输出是：

suhosin.post.max_vars = 1000  #限制最多有1000个POST参数

看PHP版本信息为：

$ php -v
PHP 5.3.2-1ubuntu4.11 with Suhosin-Patch (cli) (built: Dec 13 2011 18:45:32) 
Copyright (c) 1997-2009 The PHP Group
Zend Engine v2.3.0, Copyright (c) 1998-2010 Zend Technologies
    with Suhosin v0.9.29, Copyright (c) 2007, by SektionEins GmbH

注：测试环境是Ubuntu 10.04 + PHP 5.3.2 + PHP5-FPM

refs:
PHP哈希冲突浅析
 suhosin
Testing vs the hash collision vulnerability

Tags: Linux, PHP
没有评论

Python处理Last-Modified

Liwen — Tue, 15 Nov 2011 18:19:00 +0000

Python检查某个URI是否有修改，可以检查HTTP Response的Last-Modified字段

import urllib2
import datetime
from urllib2 import HTTPError
import datetime
def check_modified(url, last_modified, field_name="Last-Modified"):
    request = urllib2.Request(url) 
    opener = urllib2.build_opener() 
    request.add_header('If-Modified-Since', last_modified)
    try:
        resp = opener.open(request)
        print "Last-Modified: " + resp.headers.get(field_name)
    except HTTPError as e:
        if 304 == e.code:
            print "Not Modified Since: " + last_modified
 
last_modified = datetime.datetime(2011,11,15,0,0).strftime("%a, %d %h %Y %H:%M:%S GMT")
check_modified('http://www.kuaishubao.com', last_modified)

有的服务器并不返回Last-Modified, 所以需要看情况处理，比如

check_modified('http://sunliwen.com', last_modified, field_name="date")

参考：http://www.faqs.org/rfcs/rfc2616.html

Tags: HTTP, Python
没有评论

如何检测浏览器和操作系统的信息

Liwen — Thu, 10 Nov 2011 19:17:53 +0000

在实现一个功能时，需从UserAgent中抽取出浏览器和操作系统的信息。

浏览器：

Opera
Chrome
Internet Explorer
Safari
Firefox

操作系统：

Windows
iPad
iPhone
Android
Mac OS X
Linux

实现方法如下：

function getBrowser() {
    var agent = navigator.userAgent;
    return window.opera ? "Opera"
        : /chrom/i.test(agent) ? "Chrome" 
        : /msie/i.test(agent) ? "Internet Explorer" 
        : /applewebkit/i.test(navigator.appVersion) ? "Safari" 
        : /mozilla/i.test(agent) && !/compatible|webkit/i.test(agent) ? "Firefox" 
        : "";
}
 
function getOS() {
    var agent = navigator.userAgent;
    return /windows/i.test(agent) ? "Windows" 
        : /ipad/i.test(agent) ? "iPad" 
        : /iphone/i.test(agent) ? "iPhone" 
        : /android/i.test(agent) ? "Android" 
        : /mac/i.test(agent) ? "Mac OS X" 
        : /x11/i.test(agent) || /linux/i.test(agent) ? "Linux" 
        : "";
}

需要注意的是，检测顺序敏感。

参考：
* Browser detect
* Detect Mobile Browsers

Tags: Javascript
没有评论

用Python Decorator简化代码

Liwen — Fri, 14 Oct 2011 06:39:20 +0000

看码不说话。

def deco(callable):
    def method(a, b):
        print "a: %s" % a
        print "b: %s" % b
        return callable(a, b)
    return method
 
@deco
def test1(a,b):
    print a+b
 
test1(10,20)

上面的Decorator只能装饰有两个参数的函数，否则

@deco
def test2(a,b,c):
    print a+b+c

会报错

>>> test2(10,20,30)
Traceback (most recent call last):
  File "", line 1, in <module>
TypeError: method() takes exactly 2 arguments (3 given)

如需支持装饰任意参数的函数，得使用以下方法：

def deco(callable):
    def method(*args, **kws):
        print "args: %s" % repr(args)
        print "kws: %s" % repr(kws)
        return callable(*args, **kws)
    return method
 
@deco
def test1(a,b):
    print a+b
 
@deco
def test2(a,b,c):
    print a+b+c
 
test1(10,20)
test1(10,b=20)
test2(10,20,30)
test2(10,b=20,c=30)

结果是：

>>> test1(10,20)
args: (10, 20)
args: {}
30
>>> test1(10,b=20)
args: (10,)
args: {'b': 20}
30
>>> test2(10,20,30)
args: (10, 20, 30)
args: {}
60
>>> test2(10,b=20,c=30)
args: (10,)
args: {'c': 30, 'b': 20}
60

Tags: Python
没有评论

在MongoDB中sum某个字段

Liwen — Sun, 09 Oct 2011 08:26:54 +0000

假设每一条记录有一个叫做pv的整数字段，求整个集合中这个字段的和可以用如下方法:

> use test
switched to db test
> db.statistics.insert({created_on: "2011-10-01", pv: 100})
> db.statistics.insert({created_on: "2011-10-02", pv: 200})
> db.statistics.insert({created_on: "2011-10-03", pv: 300})
> var reduce = function(key, values){
...     var count=0;
...     values.forEach(function(v) {
...        count+=v.pv;
...     });
...     return {count: count};
... };
> var s = db.statistics.find();
> reduce("total_pv", s);

结果如下:

{ "count" : 600 }

Tags: MapReduce, MongoDB, NoSQL
没有评论

Nginx将所有请求转向同一个host

Liwen — Mon, 26 Sep 2011 05:38:38 +0000

准备只保留 (sunliwen.com) 这一个域名，但因为有些外链，所以加一条rewrite来兼容。

# Redirect all requests to sunliwen.com
if ($host != sunliwen.com) {
    rewrite ^(.*)$ http://sunliwen.com$1 permanent;
}

参考：

Nginx – HttpRewriteModule

Tags: Nginx, Ops
2个评论

Ubuntu下安装PHP-FPM

Liwen — Sat, 17 Sep 2011 17:28:13 +0000

把VPS的Lenny5换成了Ubuntu10.04，借这个机会试一下效率更高的 PHP-FPM 。

以下是fabric脚本：

def init_nmp():
    install_nginx()
    install_php()
    install_phpfpm()
 
def install_nginx():
    sudo("add-apt-repository ppa:nginx/stable")
    sudo("apt-get update")
    sudo("apt-get -y -q install nginx-full nginx-common")
 
def install_php():
    # More packages upon request
    sudo("apt-get -y -q --force-yes install php5-cli php5-cgi php5-mysql")
    sudo("apt-get -y -q --force-yes install php5-mcrypt libmcrypt mcrypt") 
 
def install_phpfpm():
    # TODO: use canonical php-fpm package when available
    sudo("add-apt-repository ppa:brianmercer/php")
    sudo("apt-get update")
    sudo("apt-get -y -q install php5-fpm")

在/etc/php5/fpm/php5-fpm.conf里将tcp修改成unix socket方式，单机情况下性能更好。

listen = /var/run/php5-fpm.sock
;listen = 127.0.0.1:9000

Nginx里的配置:

location ~ \.php$ { 
    try_files $uri = 404; 
    include /etc/nginx/fastcgi_params; 
    fastcgi_pass unix:/var/run/php5-fpm.sock; 
    fastcgi_index index.php; 
 
    fastcgi_split_path_info         ^(.+\.php)(.*)$; 
    include fastcgi_params; 
    fastcgi_intercept_errors        on; 
    fastcgi_ignore_client_abort     off; 
    fastcgi_connect_timeout         60; 
    fastcgi_send_timeout            180; 
    fastcgi_read_timeout            180; 
    fastcgi_buffer_size             128k; 
    fastcgi_buffers             4   256k; 
    fastcgi_busy_buffers_size       256k; 
    fastcgi_temp_file_write_size    256k; 
}

Tags: Nginx, Ops, PHP, Python, Ubuntu
没有评论