Number system with restricted set of sequence configurations
Ok, you are looking for multiple ways to represent the integers with custom symbols, with some invalid combination.
As I said in the comments, one way to do it is to make a number base. So if you have S symbols, you use the numeric base S and you are done. In this setting any zeroes to the left could be considered invalid combinations.
Now, since you want multiple difficulties, I had to think about it a bit more.
What if S is 1? Then all the combinations are the same symbol repeated multiple times. And thus the problem reduces to a mapping from natural numbers (how many times do we repeat the one symbol) to natural numbers.
Thus, I believe it is fair to split the problem in three:
How to represent integers with S symbols: This is already solved, you use the S base
How to map natural numbers to natural numbers without creating collisions and leaving invalid entries
And how to make it harder
So, for the mapping, it is easier if you think about it backwards: It is not how to come up with ways to combine the symbols so that applying some function they give you the natural numbers… Instead it is what function do you apply to the natural numbers so you get the natural numbers but skipping some, and the function can be inverted.
The solution to that is to use a linear function. So you take the natural number, multiply by some value, and add some offset. The inverted function substracts some value and divides. Not every input will give you a natural number (either because the subtraction give you a negative, or because the division is not integer), those are your invalid combinations. As as you know, the linear function does not have multiple solutions, so we know there are no collisions.
Now, to make it harder, we could take advantage of those operations… Difficulty 1: Just Substract, Difficulty 2: Just Divide, Difficulty 3: Do both.
Note: you can use prime numbers to make it harder to figure out.
However, I believe you can do more. If the goal is to make it harder to know the number, then we go to cryptography. I don’t think a substitution cipher is of much use since we are already dealing with custom symbols. But you could rearrange the digits.
What is rearranging? Well it is a mapping from the original position of the digit to the new position of the digit, so it is something similar to what we were already doing. However this time we don’t want gaps. The solution is to make blocks and add padding. Then you can do whatever permutation you want inside the block.
Now, your mapping would be:
Revert the permutations
Divide (some combinations become invalid here because they are not integer)
Substract (some combinations become invalid here because they go negative)
Change of base (if it applies)
Addendum: I’m now thinking it might be easier to change the base first and then operate, that also works.
Thus, the problem you are dealing with is: Can you do math operations with numbers that has been encrypted? And yes, that is a thing.
You might be interested in Homomorphic encryption, they do not only allow to do some computations with the encrypted values, but they allow to do those computations without knowing the values (so you can do the computation without knowing the inputs or the result).