Without Ghostscript: I Gave PDF a 'Compression Surgery' in Pure Go

Let me start with a concrete scenario.

On an intranet office computer, a document has to be submitted: the receiving system only accepts attachments under 5 MB, and a dozen-page scanned PDF in hand is 40 MB - A4 paper scanned at 300 DPI is two or three MB per page, and with enough pages it easily multiplies tenfold.

The obvious paths are all blocked. Online compression sites cannot be opened on the intranet; even if they could be, scans of reimbursement forms, contracts, and IDs should not be uploaded to an unfamiliar server. A programmer's usual answer is Ghostscript - one command. But if what you are building is a desktop tool meant to be distributed to other people, Ghostscript is an external process: a copy has to be packaged for macOS, Windows, and Kylin; the AGPL license has to be assessed; and installed intranet machines carry maintenance costs. For a tool whose installer must be self-contained, that road is far more expensive than it looks.

Image Workshop is an offline batch image tool built for this scenario (macOS / Windows / Kylin, three platforms), and 0.2.0 added PDF compression: drop a file in, click start, and get a smaller PDF. The document-submission scenario also set two bottom lines: text must not change by a single character, and images must not blur into illegibility. The implementation is pure Go - the only ready-made library in the ecosystem, pdfcpu, happens to lack the most critical piece, image recompression (an open feature request from 2017 that is still open today). So the division of labor is: pdfcpu handles parsing and writing back, and the image processing is written in-house.

Batch processing in progress on the workbench (example is an image batch). PDF needs no rule selection; drag it in and compress directly. The project is open source: github.com/Benny66/file-processor. Three-platform installers are on the Releases page.

This article covers three things: what the common way to compress PDFs in the industry is; how this implementation actually works; and what several different design choices buy compared with industry solutions.

1. How the industry compresses PDFs

First, look at what makes up PDF size. The bulk of scanned documents and mixed text-and-image documents is the embedded image stream, followed by fonts; true structural redundancy (duplicate objects, uncompressed xref tables, unused objects) is usually only a tiny fraction. So there are only three effective compression measures:

1. Lossless structural optimization: deduplication, object stream compression, cleanup of unused objects. qpdf and pdfcpu's optimize are at this level; the gains are stable but limited and cannot touch image streams.
2. Downsampling to display size: this is the biggest lever. Scans are often stored at 300-600 DPI, while screen reading only needs 150.
3. Image re-encoding: decode it, re-encode it at lower quality, and write it back.

Ghostscript packages 2 and 3 into three presets: /screen = 72 DPI, /ebook = 150 DPI, /printer = 300 DPI. It only downsamples images above the threshold, and pairs each level with a JPEG re-encode quality. Acrobat's "Reduce File Size" and the "three levels of intensity" of various online tools are, when taken apart, the same preset combinations of "DPI threshold + quality tier". This scheme is mature and effective; its only problem is that it runs in an external process and cannot be brought in.

2. Our implementation

Overall pipeline

Two terms will appear repeatedly. The xref table is the PDF object catalog, where every indirect object has an entry. XObject is a stream object referenced by the document; images live in Image XObjects, and an image with the filter name DCTDecode is JPEG bytes.

Compression is done in two steps: image-layer recompression and downsampling (implemented ourselves), then lossless structural optimization (pdfcpu's OptimizeContext: deduplicate embedded resources, compress object streams/xref streams, clean up unused objects; the content bytes stay bit-identical). Encrypted and signed files are rejected at the entrance - the former because we do not decrypt, and for the latter, changing a single byte invalidates the signature.

Embedded JPEG re-compression

Locating image objects. Instead of parsing page content streams, traverse the xref table directly; after getting each stream dictionary, use sd.Image() to tell whether it is an image object. One benefit is that an image object shared across pages is naturally processed only once.

Determining the safe subset. Only images that satisfy all of these conditions are processed:

// internal/imageproc/pdf.go
func recompressibleColorSpace(sd types.StreamDict) bool {
	cs, found := sd.Dict["ColorSpace"]
	if !found {
		return false
	}
	name, ok := cs.(types.Name)
	if !ok || (name != "DeviceRGB" && name != "DeviceGray") {
		return false // CMYK to RGB will shift colors; Indexed is a lookup-table semantics and won't match after re-encoding
	}
	bpc, ok := sd.Dict["BitsPerComponent"].(types.Integer)
	if !ok || bpc.Value() != 8 {
		return false // non-8-bit DCT is extremely rare, and the standard library can't decode it either
	}
	_, hasDecode := sd.Dict["Decode"]
	return !hasDecode // Decode is a per-component remap, so the decoded result won't match the dict declaration
}

Plus the filter must be a single DCTDecode (HasSoleFilterNamed). This predicate draws a very small circle, but each exclusion has a concrete technical reason, not a whim. Images outside the circle (Flate screenshots, fax codecs JBIG2/CCITT, CMYK, indexed color) keep their original bytes and count as "unsupported image types".

Recompression and replacement. Decode it, re-encode it with the standard library at quality 75, and give up if the encoded result is not smaller than the original:

// internal/imageproc/pdf.go
func recompressPDFImage(sd types.StreamDict, target *image.Point) (types.StreamDict, pdfImageResult) {
	if !sd.HasSoleFilterNamed(filterDCTDecode) || !recompressibleColorSpace(sd) {
		return types.StreamDict{}, pdfImageUnsupported
	}
	width, height := pdfImageDims(sd)
	if width <= 0 || height <= 0 || int64(width)*int64(height) > MaxPixels {
		return types.StreamDict{}, pdfImageUnsupported // 100 MP cap
	}
	img, _, err := image.Decode(bytes.NewReader(sd.Raw))
	if err != nil {
		return types.StreamDict{}, pdfImageUnsupported
	}
	result := pdfImageReplaced
	if target != nil && (img.Bounds().Dx() != target.X || img.Bounds().Dy() != target.Y) {
		img = resizeImage(img, target.X, target.Y) // downsample target for scanned pages
		result = pdfImageReplacedDownsampled
	}
	var out bytes.Buffer
	if err := jpeg.Encode(&out, img, &jpeg.Options{Quality: pdfJPEGQuality}); err != nil {
		return types.StreamDict{}, pdfImageUnsupported
	}
	if out.Len() >= len(sd.Raw) {
		return types.StreamDict{}, pdfImageNotSmaller // recompress not smaller: keep as-is
	}
	// ... write back
}

The quality is fixed at 75 with no multiple tiers: in the research, 75 is exactly the sweet spot of everyone's "Recommended" tier, and offering more tiers only pushes an engineering decision onto the user. A decode failure is counted as "unsupported" and skipped rather than raised as an error - one abnormal image should not ruin the whole file.

Writing back. This step has more details than expected, and missing any of them will produce a corrupt or semantically shifted PDF: replace the encoded bytes and StreamLength; if the image was scaled, restate Width/Height in the dict; restate the color space by the actual number of encoded channels (a grayscale image is still grayscale after re-encoding); remove DecodeParms, which are the original encoder's decode parameters and do not apply to a fresh bare JPEG. SMask (transparency mask) is an independent object, outside the processing scope, and is preserved as-is.

There is also a pdfcpu-specific pitfall worth a separate note: its StreamDict is a value type. What you take from an xref table entry is a copy; mutating that copy has no effect, so you must write the whole entry back:

for objNr, entry := range doc.XRefTable.Table {
	sd, ok := entry.Object.(types.StreamDict)
	if !ok || !sd.Image() {
		continue
	}
	stats.ImagesTotal++
	var target *image.Point
	if t, hit := scanTargets[objNr]; hit {
		target = &t
	}
	// ... re-encode (previous section)
	entry.Object = recompressed // value type: must write the whole entry back
}

Downsampling scanned pages

The first version only did recompression and promised not to reduce pixel dimensions. In actual use, that turned out to lock away the industry's biggest lever: scans are stored at 300-600 DPI, 150 is enough for reading on screen, and quality-only recompression cannot change the total pixel count. 0.2.0 revised that decision - a full-page scan is by default downsampled to 150 DPI, the same as Ghostscript /ebook.

The hard part is deciding whether a page is a scanned page. Downsampling any arbitrary illustration would require parsing the transformation matrices in the content stream to calculate the image's actual displayed size, which is complex and can easily hurt illustrations. Instead, the implementation avoids the content stream and takes a safe subset using structural signals:

When the condition holds, the effective DPI can be computed exactly from the page size, without touching the content stream - PDF page units are points, and 1 point = 1/72 inch:

// internal/imageproc/pdf.go, scanImageTargets
dpi := float64(imgLong) / (pageLong / 72.0)
if dpi <= pdfScanDPIFloor { // 150.0
	continue
}
scale := pdfScanDPIFloor / dpi
newLong := max(1, int(math.Round(float64(imgLong)*scale)))
newShort := max(1, int(math.Round(float64(imgShort)*scale)))

Several edge cases follow the same principle: if you are not sure, do not compress. If the aspect ratio differs by more than 15%, the image probably does not fill the page (there may be text next to it), so do not compress. If the same image is shared by multiple pages and the computed targets contradict each other, skip that image. If the DPI is only slightly above 150 and rounding coincidentally equals the original size, fall back to ordinary recompression instead of treating it as an exception.

Implementation has another pdfcpu pitfall: after ReadContext, PageCount may be 0. Iterating pages directly would execute zero times and silently return an empty result. You must call EnsurePageCount() first to complete the page tree. This kind of problem - no error, just silently not working - is not discoverable by reading the docs; it was hit and fixed while writing test fixtures that construct 300 DPI scanned pages.

3. Compared with industry solutions, where are the highlights

1. Judge each image, and replace only when smaller. Preset schemes are applied unconditionally: an image that is already well compressed gets compressed again by a preset, ending up either blurrier or larger. That is the most common failure point of online tools. Each image is encoded, compared, and if not smaller, the original bytes are put back. This check also makes running the tool again on an already-compressed file harmless: the second pass hits "already efficient" on all images and does not change a single byte.

2. DPI is computed, not estimated. Effective DPI is derived exactly from page size, without parsing content streams; scan-page detection uses structural signals (single direct reference + aspect ratio match) to take a safe subset, preferring to compress less rather than ever hurting a layout illustration. The whole downsampling pipeline has no magic numbers - 150 aligns with Ghostscript /ebook, 75 aligns with everyone's Recommended tier, and the ±15% tolerance covers the slight edge cropping common on scanned pages.

3. The result for each image is counted truthfully. The output statistics are five numbers: total images, recompressed count, count downsampled to screen sharpness, unsupported type count, and already-efficient count. Unsupported images are explicitly called out (fax codecs, CMYK, and so on - pure Go has no corresponding encoders for them), not silently skipped. In the UI, the user sees "re-saved x images, of which y were compressed to screen sharpness and z were kept unchanged" - what happened to every image is accounted for.

4. Pure Go, single binary, three platforms. The same code produces installers for macOS (Universal), Windows, and Kylin amd64/arm64 (Xinchuang cross-build), with zero external dependencies, zero CGO, and no licensing issues. As far as I know, there is currently no second open-source pure Go implementation that does object-level PDF image recompression.

5. What should not be touched is not touched; what should be rejected is rejected. Text and vector content stay byte-identical. Encrypted files are rejected. Signed files are rejected (detected by SigFlags bit0; an ordinary form that only has unsigned signature form fields is not signed and is processed normally).

4. Measured results

Item	Numbers
spike: 2400×2400 DCT image, quality 75 re-encode	whole file -42.9%, pdfcpu validate passes, rendered pixel average difference 0.41/765 (765 is the upper bound for the RGB three-channel difference, not visible to the naked eye)
300 DPI scan fixture (1200px image + 288pt square page)	downsampled to 600×600 (4 inches × 150 DPI); the test reads back the output and verifies pixels
single-file / single-image cap	512 MB / 100 MP

Test matrix: high-noise JPEG (the size difference between quality 100 and 75 is huge, and the assertion is deterministic) shrinks significantly; 300 DPI full-page image is downsampled and read back for verification; aspect-ratio mismatch is not downsampled; Flate images are counted as unsupported; encrypted and corrupt files return corresponding error codes; narrow-band falls back to ordinary recompression.

Batch summary and per-item details (example is an image batch): each row gives original size → output size and savings ratio; the summary bar gives the cumulative savings. PDF batches also report per item - "re-saved x images, of which y were compressed to screen sharpness and z were kept unchanged".

Known limitations are honestly written in the product docs: Flate/JBIG2/CCITT/JPX-encoded and CMYK/indexed-color images are kept as-is and counted; embedded fonts are not subsetted; 150 DPI is screen sharpness and may not be enough for printing.

The full implementation is in Benny66/file-processor (internal/imageproc/pdf.go, 363 lines including comments), and three-platform installers are on the Releases page. If this is useful to you, or if you have also had the idea of writing your own PDF compression in your own scenario, you are welcome to talk.