# PEP 508 Limitations

*This document collects my notes on PEP 508.*

A package manager for Python is naturally constrained by the metadata format that the
community embraces.  While a package manager can in theory add a custom metadata format
like poetry did, it limits the ability to publish such packages and prevents inter-tool
compatibility.

The metadata format in the Python ecosystem is the `pyproject.toml` and the most significant
limitation is the dependency specification from [PEP 508](https://peps.python.org/pep-0508/).

## Dependency Array Limitations

For a package manager the most important piece of information are dependencies.  In Python
there are two dependency sections, both are very limited in that they are lists of strings.

For instance `project.dependencies` is an array where each item is a PEP 508 string.  This
means that within that array, only information can be carried that PEP 508 supports.  If a
package manager requires additional information (temporary or permanently) it does not find
place in the array for it.

Example:

```toml
[project]
dependencies = [
    'Flask>=2.0',
    'more-itertools>=4.0.0,<6.0.0;python_version<="2.7"',
    'more-itertools>=4.0.0;python_version>"2.7"',
]
```

If additional information wants to be stored with either one of those dependencies, the only
reasonable correlation today would be to either duplicate it elsewhere, or to index by number.

```toml
[tool.rye.dependencies_meta.0]
extra_information = 42
```

No matter how, this is always a challenge.  In `Cargo.toml` the information is both kept as
array of objects, but additionally "merging" sections are provided.  This means that platform
and target specific dependencies are merged in at a higher level.  Additionally if a dependency
is contained more than once, it can be renamed.  For Python the `Cargo.toml` format does not make
too much sense, but some alternatives might work:

```toml
[project.dependencies]
Flask = { version = ">=2.0" }

[project.dependencies.more-itertools]
match = [
  { version=">=4.0.0,<6.0.0", python_version = "<=2.7" },
  { version=">=4.0.0", python_version = ">2.7" }
]
```

In that case at all times an object is placed in the format where extra keys could be added:

```toml
[project.dependencies]
Flask = { version = ">=2.0", extra_information = 42 }
```

That however would be a significant departure from where the format is today.  A pragmatic
alternative could be to utilize the URL portion of the dependency to funnel additional
information into the package.  Unfortunately PyPI will refuse to accept any PEP 508 marker
that contains a URL.  Additionally URLs also have restrictions in that they cannot be
combined with version markers.  The former issue might not be an issue as such a
dependency could be rewritten before publishing into an automated format.
`cargo` does something like this today already with `Cargo.toml` which is rewritten for
publishing.

Given that PEP 508 cannot really be amended without causing a massive rift in the
ecosystem the solution has to come down to encoding it into something that is already
part of the structure.  My current thinking is that the format should permit objects
where the strings are today and to re-write on publish.  So this format might become
possible:

```toml
[project]
dependencies = [
    { version = 'Flask@>=2.0', extra_information = 42 },
    'more-itertools>=4.0.0,<6.0.0;python_version<="2.7"',
    'more-itertools>=4.0.0;python_version>"2.7"',
]

# turns into

[project]
dependencies = [
    'Flask@>=2.0',
    'more-itertools>=4.0.0,<6.0.0;python_version<="2.7"',
    'more-itertools>=4.0.0;python_version>"2.7"',
]

[project.dependencies_meta.0]
exta_information = 42
```

Unfortunately that would still cause issues for tools that locally interpret `pyproject.toml`
files.

Alternatively a correlation marker could be added.  This could be done by inventing a new
marker (requires double quoting):

```toml
[project]
dependencies = [
    'Flask@>=2.0 ; "id" == "x"',
    'more-itertools>=4.0.0,<6.0.0;python_version<="2.7"',
    'more-itertools>=4.0.0;python_version>"2.7"',
]

[project.dependencies_meta.x]
extra_information = 42
```

If the correlation marker is missing, the package name would be assumed (in this case `Flask`).

## Local Dependencies

In a very similar manner local dependencies do not really work in that world yet.  Personally
I believe this is similar to the issue above as we have no reasonable way to encode both the
version and the path location into one place.  Contrast this to `Cargo.toml`:

```toml
[dependencies]
foo = { version = "1.0", path = "../crates" }
```

Upon publishing, `cargo` removes the `path` marker.  Before publishing, the `path` takes
precedence over the lookup from the index.

If one were to be able to encode a correlation marker maybe this format could work:


```toml
[project]
dependencies = [
    'Flask@>=2.0 ; "id" == "flask"',
    'more-itertools>=4.0.0,<6.0.0;python_version<="2.7"',
    'more-itertools>=4.0.0;python_version>"2.7"',
]

[project.dependencies_meta.flask]
path = "../packages/flask"
```

(Again, if there is no marker, the package name could be directly used).
