Jump to content

  • Log In with Google      Sign In   
  • Create Account


deinstancing c-strings


Old topic!
Guest, the last post of this topic is over 60 days old and at this point you may not reply in this topic. If you wish to continue this conversation start a new topic.

  • You cannot reply to this topic
48 replies to this topic

#1 fir   Members   -  Reputation: -443

Like
-1Likes
Like

Posted 20 February 2014 - 06:17 PM

is it possible (in migw) to make such things as enforcing

all identical content strings instances (like "cat", "cat", and "cat"

pplaced in three different places) to be represented in memory as

only one instance

 

i need it to compare words by adres like

 

char* str = "cat";

 

if(str=="cat") ...

 

I need whole indetical strings in my whole program to

be deisntanced to one instance - When i used old borland b55

there was a switch compiler option for this

 

also is it possible when i have the strings copies located in different 

.o modules 9i would be happy if i could deinstance that all string to

only one original instance)

 

much tnx id somebody know and could answer that

 

 



Sponsor:

#2 SunDog   Members   -  Reputation: 232

Like
1Likes
Like

Posted 20 February 2014 - 06:35 PM

Well you could do it with a constant 

 

const char* CAT_STR = "cat"

 

Also, str == "cat" is not what in C.   You want to do strcmp(str, "cat").    And it returns 0 on equality tongue.png



#3 Hodgman   Moderators   -  Reputation: 27702

Like
5Likes
Like

Posted 20 February 2014 - 06:42 PM

If you can't rely on your compiler to do this for you (which, you can't), then the process of doing it yourself is called string interning.
 
A really basic implementation might look like this (untested, just an example).

#include <set>
#include <string>

class StringDatabse
{
public:
	const char* Intern(const char* str)
	{//copy the input into the set of unique strings, return the pointer of the one in the set.
		auto result = strings.insert(str);
		return result.first->c_str();
	}
private:
	std::set<std::string> strings;
};
const char* Intern(const char* str)
{
	static StringDatabse s_strings;//P.S. This is bad. Globals are bad. Just an example...
	return s_strings.Intern(str);
}

Then you can do pointer-based string comparisons on any interned strings:

const char* str = Intern("cat");
if(str == Intern("cat"))
{
  assert( 0==strcmp(str,"cat") );
}


#4 Matias Goldberg   Crossbones+   -  Reputation: 3007

Like
3Likes
Like

Posted 20 February 2014 - 10:50 PM

Hodgman I really like, and I often support you, but that code you just posted is filled with ugliness: It's got std::set, std::string and globals. Can't get uglier than that.
 
To answer the OP post, you'll have to use const variables with extern forward declarations, and define the string in one compilation unit:
 

//Definitions.cpp
const char *gCAT = "cat";
 
//Definitions.h
extern const char *gCAT;
 
//Foo.cpp
if( strcmp( myVar, gCAT ) == 0 )
{
    //myVar contains "cat"
}

Edit: Just to clarify what's "ugly" in Hodgman's code:

  • Common implementations of std::set are not cache friendly. Any advantage from string interning is obliterated by this (duplicated string in the final compiled binary is just going to be faster).
  • std::string create unnecessary heap allocations (+ other minor misc overhead).
  • It's not thread safe (the code allows adding more strings to the pool at runtime; thus if it R/W access happens from multiple threads...)
  • It's got globals

Edited by Matias Goldberg, 20 February 2014 - 10:55 PM.


#5 fir   Members   -  Reputation: -443

Like
0Likes
Like

Posted 21 February 2014 - 01:19 AM

 

Hodgman I really like, and I often support you, but that code you just posted is filled with ugliness: It's got std::set, std::string and globals. Can't get uglier than that.
 
To answer the OP post, you'll have to use const variables with extern forward declarations, and define the string in one compilation unit:
 

//Definitions.cpp
const char *gCAT = "cat";
 
//Definitions.h
extern const char *gCAT;
 
//Foo.cpp
if( strcmp( myVar, gCAT ) == 0 )
{
    //myVar contains "cat"
}

 

this is not good solution for me, I need to compare by pointer

void DrawPixel(int x, int y, char* color)
{
    if(color=="red")  DrawPixel(x,y,0x00ff0000);
    if(color=="blue")  DrawPixel(x,y,0x000000ff);
 
}

etc it is some technique of my invention called ad-hoc enums (*)

(extremally usable becaouse you do not need to jump and at all wory

by definitions - very comfortable technique

 

in b55  i was used "unity builds" then (i mean i included 100 surce files into one then compiled it all in one 'translation unit') I got a switch "-d Merge duplicate strings" and it worked perfectly well (and i used it very heavy skiping normal enums at all)

 

Is this not possible in mingw to merge stringa across different modules ? (maybe it is possible to merge them in one translation unit?) (I think theoretically this could be quite possible, linker just need to merge many to one at a linking stage)

 

(Im very sory if i would drop my favourite ad-hoc enums in string implementation)

 

(*) precisely spiking I invented the concept of ad-hoc enum as an ad-hoc enum without definition, and found that i can use strings to implement that without dedicated compiler support

 

really it is imposible in mingw to do that?


Edited by fir, 21 February 2014 - 01:29 AM.


#6 King Mir   Members   -  Reputation: 1910

Like
2Likes
Like

Posted 21 February 2014 - 02:34 AM

gcc does merge string constants at all optimisation levels except -O0. Mingw should do likewise.

 

You can also add -fmerge-all-constants so it merges constant variables in addition to literals, but note that this is not standard conforming.


Edited by King Mir, 21 February 2014 - 02:37 AM.


#7 Hodgman   Moderators   -  Reputation: 27702

Like
3Likes
Like

Posted 21 February 2014 - 02:34 AM

Hodgman I really like, and I often support you, but that code you just posted is filled with ugliness: It's got std::set, std::string and globals. Can't get uglier than that.

Hahah no problem. It was the smallest demonstration of a string interning system that I could think of (and for the record, at least the global badness was commented).

If you want a general purpose system (e.g. dealing with strings that aren't known until runtime), then an interning system like that is what you'd want.

I've seen a bunch such systems inside other engines, except using better hash tables, and string storage containers, and lockless thread safety.... Such features aren't needed in an example that explains the idea though laugh.png

really it is imposible in mingw to do that?

It's probably possible, but I would never trust such an optimisation so much that your code's logic depends on it.
if(color=="red") is just wrong.
if(color==g_red) like in Matias' example is the only standards-compliant way to do this.



#8 fir   Members   -  Reputation: -443

Like
0Likes
Like

Posted 21 February 2014 - 04:14 AM

gcc does merge string constants at all optimisation levels except -O0. Mingw should do likewise.

 

You can also add -fmerge-all-constants so it merges constant variables in addition to literals, but note that this is not standard conforming.

 

 im not quite understand what you mean by string constants

they ale constants but i do not write a word const here

 

void DrawPixel(int x, int y, char* color)
{
    if(color=="red")  DrawPixel(x,y,0x00ff0000);
    if(color=="blue")  DrawPixel(x,y,0x000000ff);
}

 

I can change char* above to const char* but its a way to inform

compiler that "red" and "blue" in the body are also constants?

- i need youst this kind of code to work - on b55 & unity builds

(I didnt tried this with separate module strings there) it worked perfectly well (when turned -d "merge duplicate strings")

 

PS I tested it yet once (I tested before but a bit forgot the results)

 

when i use string literals in the same compiletion unit they are merged (sama adres) but when used in different compilation units

they are different (not merged) - and "-fmerge-constants -fmerge-all-constants" didnt help here I dont know why becouse I understand that it should

-fmerge-constants Attempt to merge identical constants (string constants and floating-point constants) across compilation units.

 

still not sure how to understand by constants, all string literals are considered as a constants if im not wrong so they could be safely merged imo  - but this do not work 

 

 

 

 

 

 

 

 

 

 

Edited by fir, 21 February 2014 - 05:17 AM.


#9 fir   Members   -  Reputation: -443

Like
-2Likes
Like

Posted 21 February 2014 - 04:19 AM

 

really it is imposible in mingw to do that?

It's probably possible, but I would never trust such an optimisation so much that your code's logic depends on it.
if(color=="red") is just wrong.
if(color==g_red) like in Matias' example is the only standards-compliant way to do this.

 

 

I think one could belong on that if only one know that all the "sjhsbjhs"

in his whole c program are lying in the same place - then its perfectly safe and it is not wrong - but i dont know what switch tu use to assure this

 

and I got a little breakdown because i was very accustomed to this way of writing, esp when writing roguelike projest when you use alot of enums, then i just used my string invention with great succes



#10 Aardvajk   Crossbones+   -  Reputation: 5269

Like
4Likes
Like

Posted 21 February 2014 - 05:04 AM

Just don't try to use pointers as enumeration constants. Its a bad idea - loss of enumeration type safety, hard to serialize, hard (as you have seen) to enforce the behaviour you need with different compilers, hard for other programmers to understand etc etc.

 

I totally fail to see why "red" is better than Color::red by the way. I can see reasons why Color::red is better than "red" though:

 

void f(Color::Type type);

void g(const char *type);

 

f() you can't accidentally pass a wrong type to. g() you can pass anything to. Its just abuse of the compiler and silly.

 

Yes, if you can guarantee that all of the identical strings are merged and all reside at the same address, this can work. That doesn't make it something you should do.


Edited by Aardvajk, 21 February 2014 - 05:06 AM.


#11 fir   Members   -  Reputation: -443

Like
-2Likes
Like

Posted 21 February 2014 - 05:46 AM

Just don't try to use pointers as enumeration constants. Its a bad idea - loss of enumeration type safety, hard to serialize, hard (as you have seen) to enforce the behaviour you need with different compilers, hard for other programmers to understand etc etc.

 

I totally fail to see why "red" is better than Color::red by the way. I can see reasons why Color::red is better than "red" though:

 

void f(Color::Type type);

void g(const char *type);

 

f() you can't accidentally pass a wrong type to. g() you can pass anything to. Its just abuse of the compiler and silly.

 

Yes, if you can guarantee that all of the identical strings are merged and all reside at the same address, this can work. That doesn't make it something you should do.

 

I was verry hapyy with that : you just passes string litereals everything 

works and is top efficient end easy expandable, errors and mistakes 

are also very easy to cactch here

void f(char* str)
{
       if(str=="red") ;
else if(str=="green") ;
else if(str=="orange") ;
else ERROR("string %s unrecognized in  ...", s);
}

you can also build many othet techniques on top of that (you can fall to consuming contents of the string if needed etc)

 

 

 

(thats why i am emphasizing this is my invention, (though maybe someone was used before but i never heard of it) (well two inventions really, one is a concept of "ad-hoc" enum that was first , and second is its string literal implementation)  because i was so happy with that

 

the practical reason of its great usability is the thing that you 

spare code jumping (to definition - which is tiring really),

also you do not need to maintain this enums definitions stuff

 

but aprat of this - i got a bit of breadown because this  -fmerge-constants   do not work across the modules (so i cannot use this technique across the modules) and i dont know why it does not work, and how to make it work - could someone help yet? (tnx kingmir for hint it was usefull)


Edited by fir, 21 February 2014 - 05:50 AM.


#12 Aardvajk   Crossbones+   -  Reputation: 5269

Like
3Likes
Like

Posted 21 February 2014 - 05:51 AM

I understand your reasons why you think this is a good idea. But I still maintain it isn't. You are typing all of these properties as const char*, with no way to differentiate them. Your example of the if-else chain is moving a check to runtime that should be done at compile time, so it isn't more efficient at all.

 

If the types were defined as an enumeration, the compiler can check at compile time whether a valid valid is being used (unless of course you mess about with static_casts but then all bets are off).

 

There are cases where I can see this being useful. Using strings as identifiers can reduce coupling between modules and is in some circumstances a good approach. But these cases are handled best using interning and maybe hashmapping, not by directly comparing pointers.

 

With respect, you haven't "invented" this. You have just been exploiting an implementation detail of a specific compiler in a standards-undefined manner.


Edited by Aardvajk, 21 February 2014 - 05:52 AM.


#13 fir   Members   -  Reputation: -443

Like
-4Likes
Like

Posted 21 February 2014 - 06:05 AM

I understand your reasons why you think this is a good idea. But I still maintain it isn't. You are typing all of these properties as const char*, with no way to differentiate them. Your example of the if-else chain is moving a check to runtime that should be done at compile time, so it isn't more efficient at all.

 

If the types were defined as an enumeration, the compiler can check at compile time whether a valid valid is being used (unless of course you mess about with static_casts but then all bets are off).

 

There are cases where I can see this being useful. Using strings as identifiers can reduce coupling between modules and is in some circumstances a good approach. But these cases are handled best using interning and maybe hashmapping, not by directly comparing pointers.

 

With respect, you haven't "invented" this. You have just been exploiting an implementation detail of a specific compiler in a standards-undefined manner.

 

It is very cheap check in runtime, not a problem for me (you could even include it only in debug mode)

 

I like this kind of oldschool simplicity

We got a difference of opinions here, well, go on your way if you like 


Edited by fir, 21 February 2014 - 06:10 AM.


#14 fir   Members   -  Reputation: -443

Like
0Likes
Like

Posted 21 February 2014 - 06:15 AM

 

Hodgman I really like, and I often support you, but that code you just posted is filled with ugliness: It's got std::set, std::string and globals. Can't get uglier than that.
 

 

Hodgman is a good man :) interesting answers often



#15 Aardvajk   Crossbones+   -  Reputation: 5269

Like
0Likes
Like

Posted 21 February 2014 - 06:15 AM

 

I understand your reasons why you think this is a good idea. But I still maintain it isn't. You are typing all of these properties as const char*, with no way to differentiate them. Your example of the if-else chain is moving a check to runtime that should be done at compile time, so it isn't more efficient at all.

 

If the types were defined as an enumeration, the compiler can check at compile time whether a valid valid is being used (unless of course you mess about with static_casts but then all bets are off).

 

There are cases where I can see this being useful. Using strings as identifiers can reduce coupling between modules and is in some circumstances a good approach. But these cases are handled best using interning and maybe hashmapping, not by directly comparing pointers.

 

With respect, you haven't "invented" this. You have just been exploiting an implementation detail of a specific compiler in a standards-undefined manner.

 

It is very cheap check in runtime, not a problem for me (you could even include it only in debug mode)

 

I like this kind of oldschool simplicity

We got a difference of opinions here, well, go on your way if you like 

 

 

No problem at all, each to their own. If you find a way to get this to work in a standards-defined manner across every compiler your code may every possibly be compiled with (including assurances that no future updates to the compiler will suddenly break it), please share.

 

I too would be interested in maybe using this technique if the above was not an issue.


Edited by Aardvajk, 21 February 2014 - 06:16 AM.


#16 fir   Members   -  Reputation: -443

Like
0Likes
Like

Posted 21 February 2014 - 06:22 AM



 



 



I understand your reasons why you think this is a good idea. But I still maintain it isn't. You are typing all of these properties as const char*, with no way to differentiate them. Your example of the if-else chain is moving a check to runtime that should be done at compile time, so it isn't more efficient at all.

 

If the types were defined as an enumeration, the compiler can check at compile time whether a valid valid is being used (unless of course you mess about with static_casts but then all bets are off).

 

There are cases where I can see this being useful. Using strings as identifiers can reduce coupling between modules and is in some circumstances a good approach. But these cases are handled best using interning and maybe hashmapping, not by directly comparing pointers.

 

With respect, you haven't "invented" this. You have just been exploiting an implementation detail of a specific compiler in a standards-undefined manner.

 

It is very cheap check in runtime, not a problem for me (you could even include it only in debug mode)

 

I like this kind of oldschool simplicity

We got a difference of opinions here, well, go on your way if you like 

 

 

No problem at all, each to their own. If you find a way to get this to work in a standards-defined manner across every compiler your code may every possibly be compiled with (including assurances that no future updates to the compiler will suddenly break it), please share.

 

alright, i think the assurance should be made by c standard because it is so usable, but afaik it now sadly depends on the linker 

 

in reality form me it is not primary trouble because i can stick to one given compiler (mingw) and only write code using compiler dependant 

behavior, so if it only be working here i would be happy [i know many will find write compiler dependant source (use specyfic extensions etc) as a sin but Im okay with that]



#17 Bregma   Crossbones+   -  Reputation: 4766

Like
0Likes
Like

Posted 21 February 2014 - 06:48 AM


alright, i think the assurance should be made by c standard because it is so usable, but afaik it now sadly depends on the linker

Well, the C language has assured this will not work since about 1970.  Resolving symbols and other issues regarding separate compilation units has never been a part of the language, and always a part of the system object linker.

 

If you really want to ensure the starting address of arbitrary constant data sequences in memory will be the same across all compilation units, you're going to need to use language extensions to explicitly put the constants into named sections (__attribute__((section,"my_enum"))), then use a linker script to assign those sections to a fixed base address.  You will need to use the named constant addess everywhere for comparison (as in if(color==g_red).

 

Or, you could stick to using strcmp() and move on.


Stephen M. Webb
Professional Free Software Developer

#18 fir   Members   -  Reputation: -443

Like
0Likes
Like

Posted 21 February 2014 - 07:37 AM

 


alright, i think the assurance should be made by c standard because it is so usable, but afaik it now sadly depends on the linker

Well, the C language has assured this will not work since about 1970.  Resolving symbols and other issues regarding separate compilation units has never been a part of the language, and always a part of the system object linker.

 

If you really want to ensure the starting address of arbitrary constant data sequences in memory will be the same across all compilation units, you're going to need to use language extensions to explicitly put the constants into named sections (__attribute__((section,"my_enum"))), then use a linker script to assign those sections to a fixed base address.  You will need to use the named constant addess everywhere for comparison (as in if(color==g_red).

 

Or, you could stick to using strcmp() and move on.

 

 

I do not understant why 

 

"-fmerge-constants

 

Attempt to merge identical constants (string constants and floating-point constants) across compilation units."

 

does not work- i understand it that it should work, this is probably easy by linker  to realloc all multiplicated string literal instances into one place in the data (constants probably) section - so why this option is not working, and what "attempt" mean?

 

(i would like to write a improved c dialect compiler and linker myself but is a matter of years, so by now i would like to prefer it would just be working here in mingw)

 

((customized strcmp is probably the way  to consider if primary way would be not working but its sad))


Edited by fir, 21 February 2014 - 08:00 AM.


#19 King Mir   Members   -  Reputation: 1910

Like
0Likes
Like

Posted 21 February 2014 - 09:43 AM

For my guess at why -fmerge-constants may not be working for you: are you sure it's being properly passed as a linker setting in your build system? Post the command to link your program together.

 

But I agree with others that this is a bad idea. If you need an enum, use an enum. It's faster, safer, and clearer. Here's a few reasons why:

1)You shouldn't do at runtime what you can do at compile time. An enum is a compile time constant. A pointer is a runtime constant.

2)Char * may confuse your compiler because it could alias with anything. That means it can't reorder accesses to the char pointer with other operations involving pointers. Since reordering is crucial to optimisation, this can be a major blow in non obvious ways.

3)Is it "turquose" or "turquoise" or what? If you misspell a word, it's a different value, and there's nothing to catch your mistake. Likewise, is cyan a different color or not?

4)The use of standard patterns, like enums, makes your code easier to read by others and by you in the future. Being clever is a bad thing for this.

5)If the value is not a compile time literal, your method will check it against all constant pointers and always fail to match. That's an overhead for nothing. And there's no compile time check to guarantee the pointer passes is a literal. In contrast, an enum can match with a runtime integer, or optionally not try to match a runtime integer because it's a different type.

6)On a 64 bit system, pointers are 64 bits. But an enum can have a smaller memory and register footprint. This can also mean that more function parameters are passed by register.

7)You're relying on behaviour not guaranteed by the standard. That makes your code not portable and technically not C.


Edited by King Mir, 21 February 2014 - 09:48 AM.


#20 fir   Members   -  Reputation: -443

Like
0Likes
Like

Posted 21 February 2014 - 10:07 AM

For my guess at why -fmerge-constants may not be working for you: are you sure it's being properly passed as a linker setting in your build system? Post the command to link your program together.

 

 

 

I used this way

 

c:\mingw\bin\g++ -O3 module1.c -c -fno-rtti -fno-exceptions -fmerge-constants -fmerge-all-constants

 

to compile modules (only change the name of modules here

and

 

c:\mingw\bin\g++ -O3 -Wl,--subsystem,windows -w module1.o module2.o module3.o  -lgdi32 -s -fno-rtti -fno-exceptions -o program.exe -fmerge-constants -fmerge-all-constants
 
to link them
 
i found some example by some man (in SO) when searching google about this merging (no need to write it yourself)
 
// s.c
#include <stdio.h>

void f();

int main() {
printf( "%p\n", "foo" );
printf( "%p\n", "foo" );
f();
}

// s2.c
#include <stdio.h>

void f() {
printf( "%p\n", "foo" );
printf( "%p\n", "foo" );
}
when compiled as:
gcc s.c s2.c
produces:
00403024
00403024
0040302C
0040302C

but didnt find if it is possible to merge.. this example i was not yet tried, will check this rught now 

 

edit: tested

 

sadly 
with 
 
c:\mingw\bin\gcc test.c test2.c -fmerge-constants -fmerge-all-constants
 
still got 
 
00403024
00403024
0040302C
0040302C
 
 

Edited by fir, 21 February 2014 - 10:16 AM.





Old topic!
Guest, the last post of this topic is over 60 days old and at this point you may not reply in this topic. If you wish to continue this conversation start a new topic.



PARTNERS