civilpy.general package

Submodules

civilpy.general.database_tools module

Database connection helpers.

Opens SSH-tunnelled connections to remote PostgreSQL databases (via sshtunnel and psycopg) and runs parameterized queries into pandas DataFrames. Requires the db extra: pip install civilpy[db].

civilpy.general.database_tools.ssh_into_postgres(creds)[source]

Function to open an ssh tunnel directly to a postgres database to gather data from it

Parameters:: creds – dictionary of necessary parameters to connect to the database
Returns:

civilpy.general.jupyter module

Jupyter notebook export utilities.

Wraps nbconvert to render notebooks to PDF/WebPDF/HTML with cell-tag filtering (remove_cell, remove_input, remove_output), so calculation notebooks can be published as clean reports.

civilpy.general.jupyter.notebook_converter(notebook_path, format='webpdf', text_width='70ch', branding=None)[source]

Export a notebook, limiting rendered markdown to a readable line length.

text_width caps the measure of markdown text in HTML-based exports (webpdf) so paragraphs read like an article instead of spanning the full page; code cells keep the full width. Pass None to disable. Ignored for the latex-based ‘pdf’ and ‘latex’ formats.

branding='odot' styles webpdf exports with the ODOT palette (headings in brand colors) and prepends a title block derived from the filename — unless the notebook already carries a stamped title-block cell, which is kept as-is.

civilpy.general.photos module

Site-photo utilities built on Pillow EXIF data.

Extracts EXIF metadata (including GPS coordinates and timestamps) from inspection photos, plots photo locations on maps, batch-renames files from spreadsheet lists, resizes images, and stamps dates onto photos.

class civilpy.general.photos.CivImageMap(file_path, output='show')[source]

Bases: Image

__init__(file_path, output='show')[source]

Civil Engineering Image class to display one or multiple photos on a map, multiple input and output options available

Parameters:: file_path

build_image_list_from_path(path=None)[source]

get_exif()[source]

get_geo()[source]

civilpy.general.photos.add_timestamp(image, date)[source]

Adds a timestamp with date and time extracted from an image’s metadata (if available) to the bottom right of the image.

Parameters:: image (PIL.Image.Image) – An Image object.
Returns:: The modified image with the timestamp.
Return type:: PIL.Image.Image

civilpy.general.photos.convert_filenames_from_excel(excel_file=None, root_folder=None, project_name=None, keep_existing_fns=False)[source]

Function that copies a series of file paths from column ‘A’ of an Excel file (sheet 0, with no header) and changes the file names to the new filename values held in column ‘B’. The function uses Django’s slugify function to remove illegal or confusing characters like ‘/’, ‘’, ‘,’, or ‘#’

Parameters:

excel_file (str - Path to Excel file containing file paths and new file names)
root_folder (str - Path to the folder the Renamed_Photos folder is located)
project_name (str - Name of the project to be included within the photo names)
keep_existing_fns (bool - Option to either append the original file names to the new name or not)

Return type:

None - creates the files within the new renamed folder

civilpy.general.photos.create_folder_for_renamed_files(path=None)[source]

Creates a folder named “Renamed_Photos” to store the new photos in

Parameters:: path (str - A string representation of the root folder for photos where you want the "Renamed_Photos" folder to be) – created
Return type:: None - Creates a folder in the designated location

civilpy.general.photos.get_list_of_files_from_folder(path='C:\\Users\\dane\\Desktop\\test_photos')[source]

Builds a list of files given a root folder, can be given a folder with multiple sub-folders

Parameters:: path (str - A string representation of a file path, for windows paths, either has to be passed in as a raw string,) – or with escaped/converted ‘’ characters. Once inside the function, the string is converted using the pathlib Path() module, eliminating further issues with windows paths.
Returns:: return_list
Return type:: list - A full list of every file found by the function

civilpy.general.photos.get_photo_creation_date(img)[source]

Extracts the creation date of a photo from its EXIF metadata and returns it in MM/DD/YYYY format.

Parameters:: file_path – Path to the image file
Returns:: Creation date in MM/DD/YYYY format, or None if not available

civilpy.general.photos.get_photos_from_file_list(file_list=None)[source]

Takes a list of file paths and uses regex to only return files with photo related file types (svg, heif, bmp, tiff, webp, jpeg, png, jpg). Not case-sensitive.

Parameters:: file_list (list - a list of files to be filtered for Photos)
Returns:: photos
Return type:: list - a filtered list containing only the paths from the original list that are photos

civilpy.general.photos.photo_renaming_tool(root_folder=None, excel_file=None, project_name=None)[source]

Function to simplify the above functions, given a root folder, returns a list of files and searches for an Excel file to use with the renaming tool function

Parameters:

root_folder (str - Path to folder containing photos and Excel file defining new names)
excel_file (str - Path override for if the function can't find the correct file or if it's) – in a different folder
project_name (str - Name of the project to be included in the photo names)

Return type:

None - prints list of photo files found and renames them into another folder

civilpy.general.photos.resize_image(img, width=1024, height=768, bar_color=(0, 0, 0))[source]

Resize an image while preserving the aspect ratio and add vertical (or horizontal) bars to fit the image in the specified width and height.

Parameters:

img (PIL.Image) – Path to the input image.
width (int) – Target width of the final image.
height (int) – Target height of the final image.
bar_color (tuple) – RGB color for the bars (default is black).

civilpy.general.photos.slugify(value, allow_unicode=False)[source]

Taken from https://github.com/django/django/blob/master/django/utils/text.py Convert to ASCII if ‘allow_unicode’ is False. Convert spaces or repeated dashes to single dashes. Remove characters that aren’t alphanumerics, underscores, or hyphens. Convert to lowercase. Also strip leading and trailing whitespace, dashes, and underscores.

Parameters:

value (str - the original value containing potentially illegal characters)
allow_unicode (bool - Defaults to False, prevents things like language characters and emojis from being used in) – file names

Return type:

str - a string stripped of all illegal characters for use in file names

civilpy.general.pdf_reflow module

Reflow scrambled text from MicroStation / ORD / OBM plan-set PDFs.

Bentley products emit PDF text in draw order, not reading order: lines inside a text block come out bottom-up, blocks come out in whatever order they were drawn, and multi-column sheets interleave. get_text() on a General Notes sheet is word salad — but the geometry is intact (every word carries correct coordinates), so reading order is mechanically recoverable with no OCR and no ML:

take the page’s text blocks,
cluster blocks into columns by x-position,
read columns left→right, blocks top→down, and re-sort the lines inside every block by y (undoing the bottom-up emission).

Verified against a 2026 ODOT full plan set: stream order gives "SPAN =24'-0"M AX." fragments; reflow returns the notes as written (“17. NOISE BARRIER FOUNDATION IN POOR SOIL: IN AREAS WHERE …”).

Two caveats this module is honest about:

Sheets whose text was plotted as vector curves (common for title sheets) have no text layer at all — page_has_text returns False and reflow returns "". Those pages need raster + OCR (civilpy.state.ohio.DOT.title_sheet finds the regions to OCR).
Reflow recovers paragraph text (notes, specifications). Dimension strings and callouts scattered around a detail view have no linear reading order to recover; use the words + coordinates directly for those (page.get_text("words")).

import fitz
from civilpy.general.pdf_reflow import reflow_page, extract_numbered_notes

doc = fitz.open("plan_set.pdf")
text = reflow_page(doc[99])
notes = extract_numbered_notes(text)
notes[0]     # {"num": "17", "title": "NOISE BARRIER FOUNDATION ...", ...}

civilpy.general.pdf_reflow.extract_numbered_notes(text)[source]

Split reflowed notes text into individual notes.

Returns [{"num", "title", "body"}] for headings shaped like 17. NOISE BARRIER FOUNDATION IN POOR SOIL: — the unit the non-standard-note detector compares against the standard-note library.

civilpy.general.pdf_reflow.find_notes_pages(source, pattern='GENERAL\\s+NOTES?|NOTES?\\s*:')[source]: 1-based page numbers whose text mentions a notes heading — a cheap router for where to run extract_numbered_notes().

civilpy.general.pdf_reflow.page_has_text(page, min_chars=20)[source]

Whether the page carries a usable text layer.

Bentley title sheets are often pure vector geometry (text plotted as curves) — those need the OCR path, not reflow.

civilpy.general.pdf_reflow.reflow_page(page, col_gap_frac=0.18)[source]

The page’s text in reading order (columns left→right, blocks top→down, lines within a block re-sorted by y).

Returns "" for pages with no text layer.

civilpy.general.pdf_reflow.reflow_pdf(source, pages=None, col_gap_frac=0.18)[source]

Reflow a whole document -> {page_number: text} (1-based).

source is a path or an open fitz.Document; pages limits to an iterable of 1-based page numbers. Pages without a text layer map to "" so callers can route them to OCR.

civilpy.general.pdf_ua module

Retrofit CAD drawing PDFs with a minimal PDF/UA (tagged PDF) structure.

CAD PDF exporters (Rhino’s FilePdf, Bentley Print Organizer, AutoCAD plot drivers) emit untagged PDFs: raw vectors and text runs with no /StructTreeRoot, no marked content, no alternative text. Assistive technology gets nothing, and accessibility standards (WCAG / Section 508 via PDF/UA, ISO 14289) are normally satisfied afterward by hand in Acrobat, sheet by sheet.

This module implements the honest, tractable slice of the problem — the “whole sheet is one figure” retrofit (Tier 1 in docs/Accessible_Drawings.md):

each page’s entire content stream is wrapped in a single marked-content sequence (/Figure <</MCID 0>> BDC … EMC) by prepending and appending tiny streams — the exporter’s own bytes are never modified;
one Figure structure element per page carries the alt text, a /BBox layout attribute, and is wired through a Document root and the parent tree;
the document gets the rest of the PDF/UA plumbing: /MarkInfo, /Lang, /ViewerPreferences /DisplayDocTitle, page /Tabs, a doc-info title, and XMP metadata declaring pdfuaid:part=1.

What it deliberately does not do: semantic decomposition of the drawing (per-detail figures, notes as live text, tables) — that requires regenerating the sheet, not retrofitting it (Tier 2) — and it cannot fix non-embedded fonts; those are reported as warnings instead.

from civilpy.general.pdf_ua import SheetManifest, tag_drawing_pdf

manifest = SheetManifest(
    title="DS-1-92 Drip Strip Details",
    alt_texts=("Standard drawing of a concrete drip strip: section, "
               "plan, and installation dimensions.",),
)
report = tag_drawing_pdf("DS-1-92.pdf", "DS-1-92_tagged.pdf", manifest)
for w in report.warnings:
    print(w)

Requires pikepdf (pip install civilpy[pdf]). Output should still be checked with a real validator (veraPDF, PAC) — this module makes the file structurally conformant; only a validator plus human judgment on the alt text makes it actually accessible.

class civilpy.general.pdf_ua.SheetManifest(title: str, alt_texts: Tuple[str, ...], language: str = 'en-US', author: str | None = None, subject: str | None = None, keywords: str | None = None)[source]

Bases: object

What the tagger needs to know about a drawing PDF.

alt_texts holds one entry per page, or a single entry that is broadcast to every page. The JSON form (pages as a list of objects) is the contract for anything upstream that generates manifests — per-page objects can grow fields later (finer-grained tags, reading order) without breaking Tier 1 consumers.

alt_texts: Tuple[str, ...]

author: str | None = None

classmethod from_json(path: str | Path) → SheetManifest[source]

keywords: str | None = None

language: str = 'en-US'

subject: str | None = None

title: str

to_json(path: str | Path) → None[source]

class civilpy.general.pdf_ua.TagReport(src: str, dst: str, pages: int, warnings: Tuple[str, ...]=<factory>)[source]

Bases: object

What tag_drawing_pdf() did, and what it couldn’t fix.

dst: str

pages: int

src: str

warnings: Tuple[str, ...]

Write dst as a tagged copy of src (one Figure per page).

Pass either a SheetManifest, or title= + alt_text= for the single-alt quick path. Refuses PDFs that already carry a /StructTreeRoot — stacking a second structure tree on top of an existing one produces nonsense, and un-tagging is out of scope.

Module contents

Shared helpers used across CivilPy.

The most important export is units, the shared pint.UnitRegistry that most CivilPy modules use for dimensioned quantities:

from civilpy.general import units
moment = 150 * units('kip * ft')

Also provides terminal color codes (PrintColors), simple database-to-DataFrame helpers, and unit-conversion convenience functions.

Note that Pint quantities from different registries cannot be combined arithmetically — when working with a module that builds its own registry, import units from that module instead.

class civilpy.general.PrintColors[source]

Bases: object

BOLD = '\x1b[1m'

ENDC = '\x1b[0m'

FAIL = '\x1b[91m'

HEADER = '\x1b[95m'

OKBLUE = '\x1b[94m'

OKCYAN = '\x1b[96m'

OKGREEN = '\x1b[92m'

UNDERLINE = '\x1b[4m'

WARNING = '\x1b[93m'

civilpy.general.get_table_as_df(conn, schema, table)[source]

civilpy.general package

Subpackages

Submodules

civilpy.general.database_tools module

civilpy.general.jupyter module

civilpy.general.photos module

civilpy.general.pdf_reflow module

civilpy.general.pdf_ua module

Module contents