Chapter 149: Offsets and Locks

Two forty in the morning. The dorm building had gone completely quiet. Only the water dispenser at the far end of the corridor occasionally made a low gurgling sound as it heated. Lin Chen stared at the architecture diagram on the screen. The cursor blinked over the “inter-process communication” module like a heartbeat.

In Python 2.4, the GIL was a wall he could not go around. os.fork could duplicate processes, but pipe-based communication between parent and child processes would block easily once the data volume grew large. He needed to cut two million lines of logs into chunks of fifty thousand lines each, let child processes parse them independently, and send the results back. The memory peak had to stay under control. He opened a new text file and began sketching the data flow. The parent process would distribute file offsets. Child processes would read, run regex matches, classify status codes, and write to temporary files. At the end, the parent process would merge the output in order. No shared memory—that would avoid the compatibility traps of the multiprocessing module under 2.4.

He typed the first draft of the pseudocode. def worker(pipe, offset, chunk_size): The logic was clean, but the devil lived in the details. The default pipe buffer was only 4 KB; under high concurrency, writes would hang. He would have to add non-blocking writes in the child process, or switch to direct mapping through file descriptors. He pulled out the Linux Systems Programming manual the training camp had issued them, the paper copy with a cracked spine. Chapter 14, Pipes and FIFOs. He read it word by word, his fingers unconsciously tapping the desktop. His left foot twitched again inside his slipper, the muscle stiff as a block of weathered wood. He stood up, braced himself on the edge of the desk, and slowly took two steps. As blood flowed back, a dense tingling pain spread through him. He sat down and kept going.

Replace the pipe with temporary files plus file locks. fcntl.flock. Supported in 2.4. Child processes would write into temporary files, and the parent process would read and merge them in sequence. Memory usage would drop from linear growth to something closer to constant space. He opened a terminal, used dd to generate a simulated log file with five hundred thousand lines, and ran the prototype once. The terminal scrolled; CPU usage shot instantly to eighty-five percent. Three seconds later, the result came out. He pulled up the system monitor. Peak memory usage: 21.4 MB. Runtime: 4.1 seconds. By linear estimate, two million lines would finish in around sixteen seconds, and memory would stay under 28 MB. Far below the red line of 50 MB. Engineer Li’s requirement was three hundred milliseconds per ten thousand lines, which meant total runtime had to stay within sixty seconds. Sixteen seconds looked safe, but a real production environment included network I/O and disk reads and writes. He needed more margin.

He stared at the logs streaming past in the terminal. The bottleneck was file-lock contention. Four child processes were all trying to grab the lock at the same time; context switches were burning too many CPU cycles. He needed lower-level scheduling. Linux 2.6 supported epoll, but the development image assigned by the training camp had limited permissions and incomplete user-space libraries. In the first version of the document, he could not assume every node could jump straight to the best kernel path. He flipped to Chapter 22 of the manual, I/O multiplexing. select and poll. Compatible even with 2.4. He decided to replace the polling logic around the file locks with poll, and change the temporary files to mmap memory mapping. Fewer system calls.

The clock in the lower-right corner of the screen flipped to 05:15. Outside, the sky had turned a pale gray-white. The clouds were thick, like waterlogged cotton. He saved the document as arch_v1_draft.txt. His phone screen lit up. A text from Wang Guiying: “The electricity’s fixed. The village doctor says Xiaoman’s medicine has to be switched to a different brand—the old one’s out of stock. The new brand costs twelve yuan more per box and has to be paid up front. You’re busy over there, no need to come back.”

Lin Chen stared at the words twelve yuan more. He opened his online banking page. Balance: 1,112.4 yuan. There were a little over forty yuan left on his meal card at the training camp. He transferred three hundred yuan back home. Note: medicine. The transfer fee deducted another two yuan. His balance became 810.4. He closed the page. In the county seat, twelve yuan meant two jin of pork. In Qingshi Village, it meant half a month’s worth of salt. For him, it was a fault-tolerance branch in a script, a single try-except. He couldn’t save that money, and he couldn’t spend recklessly either. He had to get the project bonus. Difficulty-A projects at the training camp came with an extra performance pool from the partner company. Engineer Li hadn’t said so explicitly, but people on the edge of the list could only turn things around that way. Rankings were fixed. Projects were alive.

Eight in the morning. The training camp computer lab. The outside unit of the air conditioner droned beyond the window. Lin Chen inserted his USB drive into a shared computer and submitted the architecture document through the internal network. The system displayed: “Received.” He sat down in front of a spare machine in the corner and waited for a response. The sound of keyboards was sparse. Several students from key provincial universities were discussing a weekend dinner gathering. Lin Chen did not look up. He kept his eyes on the time in the lower-right corner of the screen. At 9:10, a message from Engineer Li popped up on the intranet.

“I’ve read the document. The direction is right, but file locks will contend under concurrent writes. Why didn’t you use epoll in Linux 2.6?”

Lin Chen rested his fingers on the keyboard. The production server kernel was 2.6.18, but the development image assigned by the training camp was older and constrained. He could not assume the upgraded path before permission was granted. He replied: “The production server kernel is 2.6.18, but the development machine assigned by the training camp is 2.4. For compatibility, I’m temporarily using poll polling plus file locks. If a kernel upgrade is allowed, this can be replaced with edge-triggered epoll to reduce context switching.”

Engineer Li replied almost immediately: “The environment can be changed. Wednesday, two p.m., bring me a working prototype. Run two million lines. Keep latency under three hundred milliseconds. If you can’t make it, someone else gets the project.”

The message froze on the screen. Lin Chen knew it was an ultimatum. He pulled out the USB drive and stood up. When his left foot touched the floor, his knee gave way for an instant; he quickly caught himself with his right hand on the corner of the desk. A dull pain climbed from his calf through the gaps in his bones, like rusted gears grinding against each other by force. He walked slowly back to the dorm. In the afternoon, he would have to rewrite the core parsing module—replace poll with something lower-level, or simply write the extension in C. He needed to look up the Python 2.4 C API documentation. The computer section on the third floor of the library had a copy of Python Source Code Analysis, but the borrowing record showed that a graduate student from Provincial Polytechnic had checked it out last week.

Standing downstairs beneath the dormitory, he looked up at the third-floor window. The wind snapped the shirts hanging from the laundry line. He pulled out his phone and sent a text to an old classmate at Provincial Polytechnic: “Brother, can you return that copy of Python Source Code Analysis from the library before Wednesday? I need it urgently.” The message sent successfully. He leaned against the wall and closed his eyes. Wednesday, two p.m. Forty-eight hours left. The pain under his foot was sharp and distinct, like the ticking of a second hand. He opened his eyes, turned, and walked toward the cafeteria. He had to eat first. He was going to write a C extension this afternoon.

The buns in the cafeteria had already gone cold, their skins a little tough. He bought two and swallowed them down with the free soup. Back in the dorm, he pulled open the drawer and found the blank notebook. He turned to a fresh page and wrote: 1. Replace file locks with mmap memory mapping; 2. Wrap the regex-matching core in a C extension; 3. Request a 2.6 kernel environment; 4. Reserve room for the medicine expense gap.

The tip of the pen paused. He drew a circle around the fourth item, then added two words beside it: verify first. An architecture diagram, no matter how polished, meant nothing if it could not run. The training camp’s rules were explicit: theory exams counted for thirty percent, practical work forty percent, enterprise-linked projects thirty percent. The first two were already fixed. The only part left that could overturn the ranking was the last one. Difficulty-A projects were something no one had dared take on before. If you took one, you either got full marks or were eliminated outright. There was no middle ground.

He did not copy the architecture draft he had already submitted that morning. Instead, he spread out the curves left by the five-hundred-thousand-line test again: 21.4 MB, 4.1 seconds, clear file-lock contention. Two million lines looked passable in theory, but once real disks, damaged blocks, and concurrent writes entered the picture, the margin would be eaten quickly. The next step was not more explanation. It was to get access to a machine that could sustain full load and compile the C extension skeleton first.

At 5:40 in the afternoon, he submitted a request for the spare machine in Lab Area B. The reason was short: project_07 prototype, Python 2.4 C API, load test required. After sending it, he checked the USB drive, the dependency list, and the hand-copied C API function table again. The time in the lower-right corner jumped to 18:20. The administrator replied: “Area B, machine 3. Temporary permission until eight tomorrow night.”

He closed the notebook and braced himself on the edge of the bed to stand. When his left foot touched the floor, it felt as if fine grains of sand were grinding inside the joints. He folded the to-do sheet into his pocket and turned toward Lab Area B. Outside the window, the sky had already darkened, and the corridor lights came on one by one. Forty-one hours and forty minutes remained until Wednesday at two in the afternoon.

He could not break.