Documentation

From Inforail
Jump to: navigation, search

Typical things you may want to document

Compilation instructions

Testing quirks

  • Prevent shooting yourself in the foot and rendering the system unusable
    • Solaris kernel mode drivers to be kept in the temporary directory (will be erased at a reboot; otherwise the system will get stuck in a boot-crash cycle)


Environment configurations

  • PyPay for Apache //see example
  • What it was written on (language version, bitness, platform)
  • What it was tested with (as above)
  • Which services it depends on
  • Which resources it depends on
    • Databases
    • User accounts
    • Files, file-permissions
    • Open ports, port forwarding
    • Routing rules
    • Environment variables
    • OS settings (number of threads per process, stack size per process, etc)


Rationale for choices

  • Why mod_wsgi instead of mod_python //always consumes POST + reference

The rationale must be emphasized in the cases when a non-mainstream decision was taken, ex:

  • Why Win-1251 instead of UTF-8 (when the best practices recommend UTF-8)

Known quirks

  • Why printing to stdout is not allowed

Maintenance instructions

  • Asterisk test calls with DTMF checks

Troubleshooting

  • Where the logs go
  • How to adjust logging verbosity
  • How to test an individual module without causing downtime
  • Verification checklist


Specs and requirements

  • PyPay schemas
  • Include copies of the standards if possible (not just a reference to it)

Exotic components

When the system relies on some rare and hard to find component, it is a good idea to keep the installer of these components, rather than just describing what was used. This can make a difference N years later, when you have to maintain a system discontinued by the manufacturer (EOL).

Development environments

  • Virtual machine image with a ready-to-go environment //Zastava

Business processes

  • Release procedure checklist
  • Internal affairs
    • When someone is hired
    • When someone is fired
    • Using the code repository //SVN, CVS, etc
  • The internal phone system
  • CMS maintenance


Examples

Setting up and maintaining PyPay

PyPay - payment processing and tracking system (c) Dekart


Dependencies

Platform

PyPay was developed and tested with the following:

  • Python 2.6.4 (r264:75708) for Windows x86
  • Python 2.5.2 (r252:60911) on Debian Lenny
  • mod_wsgi 2.5
  • Apache 2.2.9 (from Debian repositories)

It does not rely on any exotic features, so there are no reasons to believe it will not work on newer versions of these components. However, note that Python 3.x is a different language, so the program is not supposed to work in it.

Modules

PyPay depends on the following modules

  • Python MySQLdb

Services

PyPay depends on the following parts of the environment

  • mail server
  • MySQL database

For more details, see the 'Configuration' section.


Setting up the environment

Apache and mod_python

  • mod_python cannot be used to read POST requests that come in a form other than url-encoded. The mod_python publisher will always attempt to consume the POST request, thus you will never be able to see the raw data.
  • https://issues.apache.org/jira/browse/MODPYTHON-29

Apache and mod_wsgi

  • Setting up mod_wsgi - http://ubuntuforums.org/showthread.php?t=833766
  • Reading the body of a POST request via stdin
    • this must be done by indicating the exact size of the data you want to read, get it from Content-Length in the HTTP request header.
    • If you use read() with no arguments, the function will hang indefinitely on most webservers
  • Current directory of the script: http://code.google.com/p/modwsgi/wiki/ApplicationIssues
  • Writing to stdout is prohibited, i.e. do not use the print statement: http://code.google.com/p/modwsgi/wiki/ApplicationIssues
  • Importing modules is made easier by updating the system path with the path of the directory with the modules
    • sys.path.insert(0, "path/to/dir") #no trailing slash
    • scriptPath = environ['SCRIPT_FILENAME'] #this retrieves the path to the wsgi script itself
  • Debugging
    • exceptions thrown by Python can be seen in Apache's error.log, usually in /var/log/apache2/error.log
    • you can print to that file directly with
print >> environ['wsgi.errors'], "application debug A"

Configuration

The configuration of PyPay is located in PyPayconf.py, that is the single place where you need to make changes to alter:

  • the logging verbosity level
  • path to the log
  • email server settings
  • email addresses
  • database connection credentials

New features that are added must follow the same configuration principle and load their settings from this file. Do not hardcode magic values, I will catch you doing that and I'll get angry.

Debugging

Debugging can be done in several ways

  • On the server, by reading the logs of
    • PyPay
    • logs of Apache
  • On a development machine. PyPay's modules have additional sets of instructions in the __main__ module. If you run the script as a standalone entity, you will be able to test separate functions, without modifying anything in index.wsgi itself; this is really handy. PyPay will also keep a log in a different place if it detects that it is on a development machine.

Note that this detection is done by checking if the plaform Python is running on is NT. If you're developing it on Linux, alter that rule somehow.

Modus operandi

  • See 01-high-level.png to get the big picture
  • 01..03-DB.png illustrate the steps that happen inside the AddTransactionToDCD function
    • these actions were determined by reverse-engineering the old PHP script, which was undocumented, had bugs, and a horrible coding style. There is a chance that some bits were not interpreted correctly; although extensive tests were conducted to make sure that is not the case.

Extending

  • if the notifications from another processing company come via HTTP POST requests,
    • add another URL handler inside index.wsgi (ex: /acme-company) and handle it as in this example:
      • if environ['PATH_INFO'] == "/element5":
    • create another derivative class from ReportParser, inside which you will populate an object of the Transaction type using data from the XML that came to you
  • if the notification comes via HTTP GET (ex: as with Gate2Shop), handle it like this:
    • if environ['REQUEST_METHOD']!='POST':

Warning - as long as you don't change anything in the Transaction class, your extensions will not affect the existing parsers. If you make changes, be careful.

Files

  • post.py - it is used to send a POST request with some payload to a server. It comes handy when testing the parser for Element5.
  • /test-files - collection of XML notifications the parser was tested against. If the parser fails with any of them, that's bad news; always perform tests and make sure parsing is going well.
  • /doc - documentation and supporing materials
    • XML_Schema_2_2_EN.pdf - contains the description of the format of the notifications delivered by Element5
    • /model - the original PHP script used to understand what happens in DCD and replicate the same functionality. Beware, it is ugly ;-)

References

Building wxWidgets for Keeper

1. Download WxWidgets 2.8.9 (this is what Keeper was tested with; adventures 
	are imminent if you tinker with other versions)
	
	If the file is not listed on the official site, go to http://sourceforge.net/projects/wxwindows/files/wxMSW/ and take the 
	specific Windows build

2. Run the installer and set it up. 
	WARNING - when compiling and building the whole thing, a lot of space will be
	required. Make sure there are ~3 GB available on the partition where the
	action happens

3. Go to \build\msw, open wx.dsw (the VC6 project)

4. Convert it to your current version of Visual Studio (tested with VS2008)

5. Enable wxUSE_STD_IOSTREAM (this is used by Keeper's logging mechanism), set
	#define wxUSE_STD_IOSTREAM 1
	
	This macro is defined in the project wxregex in setup.h, note that there are
	two files with this name, but you only need to change this in \msw\setup.h

6. Enable wxUSE_UNICODE, in \msw\setup.h, change this

	#ifndef wxUSE_UNICODE
		#define wxUSE_UNICODE 0
	#endif
	
	to
	
	#ifndef wxUSE_UNICODE
		#define wxUSE_UNICODE 1
	#endif
	

7. 	Go to Build\Batch build
	Select all, and begin building
	
	get some tea :-)
	
	Sometimes it will throw some linker errors, it is safe to ignore them. After
	the build process, run the build again - it will now be able to find the 
	LIBs it didn't have earlier. This is caused by the fact that the projects
	are not built in the right order - some libs are requested before they are
	actually available.
	
8. Repeat the build process until you get no more linker errors. Note that each 
	time it will not rebuild everything, but only the stuff that hasn't been 
	built. Unless you do a Clean, that is ;-)
	
9. Add an environment variable to the system to indicate where wxWidgets resides,
	ex: WXWIN = D:\wxWidgets-2.8.9
	
	You may have to restart Explorer, Visual Studio or the CMD interpreter before
	the variable is visible. If you're not sure what to do, just reboot the PC.
	
10. In Visual Studio's main menu, go to Tools\Options\Projects and Solutions
	Select "VC++ directories"
	Select Platform = Win32
	
	Add the following entries:
	- Include files
		$(WXWIN)/contrib/include
		$(WXWIN)/include
	- Library files
		$(WXWIN)/contrib/lib
		$(WXWIN)/lib/vc_lib