Bacteria are ubiquitous single cellular organisms. Compared to eukaryotic cells, bacteria have two unique characteristics: the membrane-less nucleoid and the cell wall built of peptidoglycan (PG). In most bacteria, a single circular chromosome is compacted in a small volume (2-5 µm in length and ~1 µm in width). It is intriguing how bacteria can accomplish a rapid and faithful replication (~1000 bp/s) within such complicated environments, where other DNA-associated events like segregation, transcription, and repair take place simultaneously. On the other hand, understanding how the bacterial cell wall is synthesized and remodeled is fundamentally important, of which underlying mechanisms not only govern the cell shape and size control, but also contribute to the development of novel antibiotics. In the first part of this thesis, I investigated replisome dynamics over the cell cycle of Caulobacter crescentus. The replisome is a multiprotein machinery responsible for DNA duplication. In C. crescentus, two replisomes assemble at the origin of replication site (ori), duplicate DNA bidirectionally, and final meet at the terminus site (ter) and disassemble. By using time-lapse live-cell microscopy, I found that the replisome dynamics are flexibly shaped by the chromosome organization including chromosome segregation, inter-arm alignment, and site-specific replication-transcription conflicts. Our results addressed the long puzzling of how the sister replisomes are organized, and conciliate two opposing models -- "factory" and "tracking" model which depict two replisomes are colocalized or independent. After having focused my studies on the dynamics of bacterial replisomes, I shifted my attention to the PG synthesis of the cell wall. In the second part of this thesis, I investigated the cell wall synthesis dynamics with collaborators using both experimental and analytic tools. We first developed new fluorescent probes to visualize the nascent cell wall growth using single-molecule localization microscopy (SMLM), which enables resolving the cell wall growth pattern at nanoscale. To reconstruct the dynamic information lacked in SMLM, we developed a machine-learning based tool, to pseudo-temporally order bacterial shape dynamics over the cell cycle where only static SMLM images were used as input. Applying these tools, we finally demonstrated a successful reconstruction of nascent cell wall growth in Streptococcus pneumoniae. In summary, this thesis focused on two main questions in bacterial cell biology: replisome organization and cell wall synthesis. The results obtained and the methods developed should be broadly interesting to the community of bacterial cell biology, and inspire more fundamental studies in this field.