1
$\begingroup$

Neural networks operate on numbers, and it's well-known what the derivative of numeric functions are, as well as what the derivative of matrix functions are.

What about functions that operate on maps or strings? Is there a derivative of the union/difference of a map, or the append/replace/delete/concat of a string? Does the chain rule still apply, or does another chain rule now exist?

What, if any, would be the study of the calculus of sets and strings? Would it be a calculus of sets and ordered sets?

Improve this question
$\endgroup$
5
  • 1
    $\begingroup$ Because you’ve framed this question in terms of neural networks, I’m obliged to point out that NNs don’t typically operate on strings themselves, but instead map strings to floating point numbers (using embedding layers) and then proceed in the ordinary way for numerical data. I’m unfamiliar with any neural network that does otherwise. $\endgroup$ Commented Feb 17, 2023 at 22:30
  • $\begingroup$ I’ve voted to move this question to Math.SE as it seems to be about a theory of derivatives of arbitrary mappings, rather than being about statistics. $\endgroup$ Commented Feb 17, 2023 at 22:34
  • 3
    $\begingroup$ There are two (closely related) concepts of derivatives. One is that of studying the response of a function when its arguments are changed only a little. That's not possible with discrete arguments. The other is an algebraic definition of a derivation, but that requires some kind of linear structure (vector space or its generalization). $\endgroup$ Commented Feb 17, 2023 at 22:47
  • $\begingroup$ @Dave Sure, if that works. How will I know when it has moved? Or will I just have to x-post it over at math.SE? $\endgroup$ Commented Feb 17, 2023 at 23:40
  • $\begingroup$ I would suggest first researching related posts on Mathematics and then, if you still don't find what you're looking for, ask your question there (perhaps as modified by any new information you have gleaned, such as offered in the comment thread here). In particular, the reference to NNs might obfuscate the question more than it helps with the audience there. $\endgroup$ Commented Feb 18, 2023 at 13:49

1 Answer 1

Reset to default
1
$\begingroup$

With adequate care you might find a derivation for your operator which generalizes derivatives in an algebraic way, but functions over the elements of discrete structures such as strings and sets do not have derivatives.

Another approach is to introduce some functionals of metrics, as I briefly do below.

If you define a metric $d$ on some structures $\Omega$ and $\Omega^{\prime}$, you could consider the pairwise metric dilation

$$\operatorname{dil}_{\operatorname{pairwise}}(\eta) \triangleq \frac{d (\eta(u), \eta(v))}{d (u,v)}$$

or pairwise metric distortion

$$\operatorname{dis}_{\operatorname{pairwise}}(\eta) \triangleq |d (\eta(u), \eta(v)) - d (u,v)|$$

for a map $\eta:\Omega \mapsto \Omega^{\prime}$ and $u,v \in \Omega$.

Similar forms of above appear in Bronstein et al 2021, which take the supremum over possible choices of $u,v$ where $u \neq v$. They also have a invoke further generality of considering possibly-different metrics over $\Omega$ and $\Omega^{\prime}$.

Improve this answer
$\endgroup$
3
  • $\begingroup$ I've been remiss in addressing the neural network part of the question. I will return to it later if I can. $\endgroup$ Commented Feb 17, 2023 at 22:34
  • $\begingroup$ By metrics, I assume you mean something like Jaccard distance or Levenshtein distance? But in a distance measure, the metric doesn't uniquely reconstruct Ω′ from Ω (as I understand it). If that's the case, are there other things that, with the "derivatives" of a sequence of operations can be compressed together with a chain rule to get you from Ω to Ω′? $\endgroup$ Commented Feb 17, 2023 at 23:53
  • $\begingroup$ I'm not sure what you mean by "reconstruct". It is true that you cannot use metric dilation/distortion alone to get the function $\eta$, although the way that fails depends on the symmetries of the metric you chose. Derivatives by themselves don't reconstruct a function either. Even FTC only lets you find an antiderivative up to a constant, if integration is what you meant by "reconstruct". $\endgroup$ Commented Feb 18, 2023 at 1:54

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.