firefox-52.8.*esr keeps crashing
I build/run a LFS system. My latest build, with gcc-7.3 by the way, of ff-52.8.0esr & ff-52.8.1esr "keeps" crashing, depending on the website--hasn't crashed right here yet . LFS doesn't use the crash-reporter option, but I have this: Jun 11 10:01:42 myhost kernel: Web Content[19060]: segfault at fffe8 ip 00007f063c961de1 sp 00007ffd4e33eab0 error 4 in libxul.so[7f0639f1e000+3e80000] Jun 11 10:03:42 myhost kernel: Web Content[26699]: segfault at fffe8 ip 00007f67f1fe7de1 sp 00007ffc9785a920 error 4 in libxul.so[7f67ef5a4000+3e80000] I've seen similar reports using gcc-[89]*, only 7.3 here. One expects esrs to be stable. Does this help?
Application Basics
Name: Firefox Version: 52.8.1 Build ID: 20180611082546 User Agent: Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101 Firefox/52.0 OS: Linux 4.14.39-SMP Multiprocess Windows: 1/1 (Enabled by default) Safe Mode: true
Extensions
Name: Application Update Service Helper Version: 2.0 Enabled: true ID: aushelper@mozilla.org
Name: Multi-process staged rollout Version: 1.10 Enabled: true ID: e10srollout@mozilla.org
Name: Pocket Version: 1.0.5 Enabled: true ID: firefox@getpocket.com
Name: Web Compat Version: 1.0 Enabled: true ID: webcompat@mozilla.org
Graphics
Features Compositing: Basic Asynchronous Pan/Zoom: wheel input enabled; touch input enabled WebGL Renderer: WebGL is currently disabled. WebGL2 Renderer: WebGL is currently disabled. Hardware H264 Decoding: No Audio Backend: alsa GPU #1 Active: Yes Description: VMware, Inc. -- Gallium 0.4 on llvmpipe (LLVM 4.0, 128 bits) Vendor ID: VMware, Inc. Device ID: Gallium 0.4 on llvmpipe (LLVM 4.0, 128 bits) Driver Version: 3.0 Mesa 17.1.6
Diagnostics AzureCanvasAccelerated: 0 AzureCanvasBackend: skia AzureContentBackend: skia AzureFallbackCanvasBackend: none CairoUseXRender: 0 Decision Log HW_COMPOSITING: blocked by default: Acceleration blocked by platform blocked by runtime: Acceleration blocked by safe-mode OPENGL_COMPOSITING: unavailable by default: Hardware compositing is disabled
Important Modified Preferences
accessibility.typeaheadfind.flashBar: 0 browser.cache.disk.capacity: 35840 browser.cache.disk.filesystem_reported: 1 browser.cache.disk.smart_size.enabled: false browser.cache.disk.smart_size.first_run: false browser.cache.disk.smart_size.use_old_max: false browser.cache.frecency_experiment: 2 browser.download.importedFromSqlite: true browser.download.useDownloadDir: false browser.places.smartBookmarksVersion: 8 browser.search.suggest.enabled: false browser.search.update: false browser.sessionstore.upgradeBackup.latestBuildID: 20180611082546 browser.startup.homepage_override.buildID: 20180611082546 browser.startup.homepage_override.mstone: 52.8.1 browser.tabs.remote.autostart.2: true browser.urlbar.autocomplete.enabled: false browser.urlbar.suggest.bookmark: false browser.urlbar.suggest.history: false browser.urlbar.suggest.openpage: false extensions.lastAppVersion: 52.8.1 media.gmp-gmpopenh264.abi: x86_64-gcc3 media.gmp-gmpopenh264.lastUpdate: 1527573251 media.gmp-gmpopenh264.version: 1.6 media.gmp-manager.buildID: 20180611082546 media.gmp-manager.lastCheck: 1528736272 media.gmp.storage.version.observed: 1 network.cookie.cookieBehavior: 1 network.cookie.prefsMigrated: true network.predictor.cleaned-up: true places.database.lastMaintenance: 1527699721 places.history.expiration.transient_current_max_pages: 122334 plugin.disable_full_page_plugin_for_types: application/pdf services.sync.declinedEngines: storage.vacuum.last.index: 0 storage.vacuum.last.places.sqlite: 1527699721
Important Locked Preferences
Places Database
JavaScript
Incremental GC: true
Accessibility
Activated: false Prevent Accessibility: 0
Library Versions
NSPR Expected minimum version: 4.16 Version in use: 4.16
NSS Expected minimum version: 3.32 Version in use: 3.32
NSSSMIME Expected minimum version: 3.32 Version in use: 3.32
NSSSSL Expected minimum version: 3.32 Version in use: 3.32
NSSUTIL Expected minimum version: 3.32 Version in use: 3.32
Experimental Features
Sandbox
Seccomp-BPF (System Call Filtering): true Seccomp Thread Synchronization: true User Namespaces: false Media Plugin Sandboxing: true
All Replies (20)
Cross-posting your crash report IDs:
paulrogers said
Please direct me to a patch for 52.8.[01]esr.
Report ID Date Submitted bp-e0275d73-1dfd-44df-bad3-8a7aa1180611 6/11/18 1:11 PM bp-f76f7d0e-3ea2-4da8-aa37-5c1b60180611 6/11/18 1:05 PM
firefox Crash Reason SIGSEGV
Check into your Video Card Drivers as part of the issue : featureLog: {u'fallbacks': [], u'features': [{u'status': u'blocked', u'description': u'Compositing', u'log': [{u'status': u'blocked', u'message': u'Acceleration blocked by platform', u'type': u'default'}], u'name': u'HW_COMPOSITING'}, {u'status': u'unavailable', u'description': u'OpenGL Compositing', u'log': [{u'status': u'unavailable', u'message': u'Hardware compositing is disabled', u'type': u'default'}], u'name': u'OPENGL_COMPOSITING'}]}
I'm not sure what's required of the video driver. I've recently built this system, so X-7.7 isn't very old. I've been building on an i7, but I'm building for a generic type system as if for a distro to be cloned. The i7 has an Nvidia video card, but I'm using the generic VESA driver as a "lowest common denominator". Here's the Xorg.log.0 (Nope, field's not big enough):
[ 2760.802] X.Org X Server 1.19.3 Release Date: 2017-03-15 [ 2760.803] X Protocol Version 11, Revision 0 [ 2760.803] Build Operating System: Linux 4.14.39-SMP x86_64 [ 2760.804] Current Operating System: Linux persia 4.14.39-SMP #7 SMP Mon May 28 22:22:07 PDT 2018 x86_64 [ 2760.804] Kernel command line: BOOT_IMAGE=(hd0,1)/boot/vmlinuz ro root=/dev/sda1 video=640x480-16 nomodeset [ 2760.804] Build Date: 19 May 2018 06:22:13PM [ 2760.805] [ 2760.805] Current version of pixman: 0.34.0 [ 2760.805] Before reporting problems, check http://wiki.x.org
to make sure that you have the latest version.
[ 2760.805] Markers: (--) probed, (**) from config file, (==) default setting,
(++) from command line, (!!) notice, (II) informational,
(WW) warning, (EE) error, (NI) not implemented, (??) unknown.
[ 2760.807] (==) Log file: "/var/log/Xorg.0.log", Time: Mon Jun 11 13:42:48 2018 [ 2760.807] (==) Using config directory: "/etc/X11/xorg.conf.d" [ 2760.808] (==) Using system config directory "/usr/share/X11/xorg.conf.d" [ 2760.808] (==) No Layout section. Using the first Screen section. [ 2760.808] (**) |-->Screen "Screen0" (0) [ 2760.808] (**) | |-->Monitor "<default monitor>" [ 2760.808] (**) | |-->Device "VESA" [ 2760.808] (**) | |-->GPUDevice "Intel Graphics" [ 2760.808] (**) | |-->GPUDevice "nvidia" [ 2760.808] (**) | |-->GPUDevice "VESA" [ 2760.808] (==) No monitor specified for screen "Screen0".
Using a default monitor configuration.
[ 2760.808] (==) Automatically adding devices [ 2760.808] (==) Automatically enabling devices [ 2760.808] (==) Automatically adding GPU devices [ 2760.808] (==) Max clients allowed: 256, resource mask: 0x1fffff [ 2760.808] (WW) `fonts.dir' not found (or not valid) in "/usr/share/fonts/X11/misc/". [ 2760.808] Entry deleted from font path. [ 2760.808] (Run 'mkfontdir' on "/usr/share/fonts/X11/misc/"). [ 2760.808] (WW) `fonts.dir' not found (or not valid) in "/usr/share/fonts/X11/100dpi/". [ 2760.808] Entry deleted from font path. [ 2760.808] (Run 'mkfontdir' on "/usr/share/fonts/X11/100dpi/"). [ 2760.808] (WW) `fonts.dir' not found (or not valid) in "/usr/share/fonts/X11/75dpi/". [ 2760.808] Entry deleted from font path. [ 2760.808] (Run 'mkfontdir' on "/usr/share/fonts/X11/75dpi/"). [ 2760.808] (==) FontPath set to:
/usr/share/fonts/X11/TTF/,
/usr/share/fonts/X11/OTF/,
/usr/share/fonts/X11/Type1/
[ 2760.808] (==) ModulePath set to "/usr/lib/xorg/modules" [ 2760.808] (II) The server relies on udev to provide the list of input devices.
If no devices become available, reconfigure udev or disable AutoAddDevices.
[ 2760.808] (II) Loader magic: 0x89cc60 [ 2760.808] (II) Module ABI versions: [ 2760.808] X.Org ANSI C Emulation: 0.4 [ 2760.808] X.Org Video Driver: 23.0 [ 2760.808] X.Org XInput driver : 24.1 [ 2760.808] X.Org Server Extension : 10.0 [ 2760.814] (--) PCI:*(0:2:0:0) 10de:0dc4:0000:0000 rev 161, Mem @ 0xd0000000/16777216, 0xe8000000/134217728, 0xf0000000/33554432, I/O @ 0x00002000/128, BIOS @ 0x????????/131072 [ 2760.814] (WW) Open ACPI failed (/var/run/acpid.socket) (No such file or directory) [ 2760.814] (II) LoadModule: "glx" [ 2760.814] (II) Loading /usr/lib/xorg/modules/extensions/libglx.so [ 2760.816] (II) Module glx: vendor="X.Org Foundation" [ 2760.816] compiled for 1.19.3, module version = 1.0.0 [ 2760.816] ABI class: X.Org Server Extension, version 10.0 [ 2760.816] (II) LoadModule: "vesa" [ 2760.816] (II) Loading /usr/lib/xorg/modules/drivers/vesa_drv.so [ 2760.816] (II) Module vesa: vendor="X.Org Foundation" [ 2760.816] compiled for 1.19.3, module version = 2.3.3 [ 2760.816] Module class: X.Org Video Driver [ 2760.816] ABI class: X.Org Video Driver, version 23.0 [ 2760.816] (II) LoadModule: "intel" [ 2760.816] (WW) Warning, couldn't open module intel [ 2760.816] (II) UnloadModule: "intel" [ 2760.816] (II) Unloading intel [ 2760.816] (EE) Failed to load module "intel" (module does not exist, 0) [ 2760.816] (II) LoadModule: "nouveau" [ 2760.816] (II) Loading /usr/lib/xorg/modules/drivers/nouveau_drv.so [ 2760.817] (II) Module nouveau: vendor="X.Org Foundation" [ 2760.817] compiled for 1.19.3, module version = 1.0.15 [ 2760.817] Module class: X.Org Video Driver [ 2760.817] ABI class: X.Org Video Driver, version 23.0 [ 2760.817] (II) VESA: driver for VESA chipsets: vesa [ 2760.817] (II) NOUVEAU driver [ 2760.817] (II) NOUVEAU driver for NVIDIA chipset families : [ 2760.817] RIVA TNT (NV04) [ 2760.817] RIVA TNT2 (NV05) [ 2760.817] GeForce 256 (NV10) [ 2760.817] GeForce 2 (NV11, NV15) [ 2760.817] GeForce 4MX (NV17, NV18) [ 2760.817] GeForce 3 (NV20) [ 2760.817] GeForce 4Ti (NV25, NV28) [ 2760.817] GeForce FX (NV3x) [ 2760.817] GeForce 6 (NV4x) [ 2760.817] GeForce 7 (G7x) [ 2760.817] GeForce 8 (G8x) [ 2760.817] GeForce GTX 200 (NVA0) [ 2760.817] GeForce GTX 400 (NVC0) [ 2760.817] (--) using VT number 7
[ 2760.827] (II) Loading sub module "vbe" [ 2760.827] (II) LoadModule: "vbe" [ 2760.827] (II) Loading /usr/lib/xorg/modules/libvbe.so [ 2760.827] (II) Module vbe: vendor="X.Org Foundation" [ 2760.827] compiled for 1.19.3, module version = 1.1.0 [ 2760.827] ABI class: X.Org Video Driver, version 23.0 [ 2760.827] (II) Loading sub module "int10" [ 2760.827] (II) LoadModule: "int10" [ 2760.827] (II) Loading /usr/lib/xorg/modules/libint10.so [ 2760.827] (II) Module int10: vendor="X.Org Foundation" [ 2760.827] compiled for 1.19.3, module version = 1.0.0 [ 2760.827] ABI class: X.Org Video Driver, version 23.0 [ 2760.827] (II) VESA(0): initializing int10 [ 2760.828] (II) VESA(0): Bad V_BIOS checksum [ 2760.828] (II) VESA(0): Primary V_BIOS segment is: 0xc000 [ 2760.899] (II) VESA(0): VESA BIOS detected [ 2760.899] (II) VESA(0): VESA VBE Version 3.0 [ 2760.899] (II) VESA(0): VESA VBE Total Mem: 14336 kB [ 2760.899] (II) VESA(0): VESA VBE OEM: NVIDIA [ 2760.899] (II) VESA(0): VESA VBE OEM Software Rev: 112.6 [ 2760.899] (II) VESA(0): VESA VBE OEM Vendor: NVIDIA Corporation [ 2760.899] (II) VESA(0): VESA VBE OEM Product: BIOS-P/N@N7087 [ 2760.899] (II) VESA(0): VESA VBE OEM Product Rev: Chip Rev [ 2761.028] (**) VESA(0): Depth 16, (--) framebuffer bpp 16 [ 2761.028] (==) VESA(0): RGB weight 565 [ 2761.028] (==) VESA(0): Default visual is TrueColor [ 2761.028] (==) VESA(0): Using gamma correction (1.0, 1.0, 1.0) [ 2761.028] (II) Loading sub module "ddc" [ 2761.028] (II) LoadModule: "ddc" [ 2761.028] (II) Module "ddc" already built-in [ 2761.030] (II) VESA(0): VESA VBE DDC supported [ 2761.030] (II) VESA(0): VESA VBE DDC Level 2 [ 2761.030] (II) VESA(0): VESA VBE DDC transfer in appr. 1 sec. [ 2761.079] (II) VESA(0): VESA VBE DDC read successfully [ 2761.079] (II) VESA(0): Manufacturer: SAM Model: 192 Serial#: 1112092983 [ 2761.079] (II) VESA(0): Year: 2005 Week: 24 [ 2761.079] (II) VESA(0): EDID Version: 1.3 [ 2761.079] (II) VESA(0): Digital Display Input [ 2761.079] (II) VESA(0): Max Image Size [cm]: horiz.: 34 vert.: 27 [ 2761.079] (II) VESA(0): Gamma: 2.20 [ 2761.079] (II) VESA(0): DPMS capabilities: Off [ 2761.079] (II) VESA(0): Supported color encodings: RGB 4:4:4 YCrCb 4:4:4 [ 2761.079] (II) VESA(0): First detailed timing is preferred mode [ 2761.079] (II) VESA(0): redX: 0.650 redY: 0.330 greenX: 0.330 greenY: 0.600 [ 2761.079] (II) VESA(0): blueX: 0.150 blueY: 0.080 whiteX: 0.313 whiteY: 0.329 [ 2761.079] (II) VESA(0): Supported established timings: [ 2761.079] (II) VESA(0): 720x400@70Hz [ 2761.079] (II) VESA(0): 640x480@60Hz [ 2761.080] (II) VESA(0): 640x480@67Hz [ 2761.080] (II) VESA(0): 640x480@72Hz [ 2761.080] (II) VESA(0): 640x480@75Hz [ 2761.080] (II) VESA(0): 800x600@56Hz [ 2761.080] (II) VESA(0): 800x600@60H[ 2761.080] (II) VESA(0): 800x600@72Hz [ 2761.080] (II) VESA(0): 800x600@75Hz [ 2761.080] (II) VESA(0): 832x624@75Hz [ 2761.080] (II) VESA(0): 1024x768@60Hz [ 2761.080] (II) VESA(0): 1024x768@70Hz [ 2761.080] (II) VESA(0): 1024x768@75Hz [ 2761.080] (II) VESA(0): 1280x1024@75Hz [ 2761.080] (II) VESA(0): 1152x864@75Hz [ 2761.080] (II) VESA(0): Manufacturer's mask: 0 [ 2761.080] (II) VESA(0): Supported standard timings: [ 2761.080] (II) VESA(0): #0: hsize: 1280 vsize 1024 refresh: 60 vid: 32897 [ 2761.080] (II) VESA(0): #1: hsize: 1280 vsize 960 refresh: 60 vid: 16513 [ 2761.080] (II) VESA(0): #2: hsize: 1152 vsize 864 refresh: 75 vid: 20337 [ 2761.080] (II) VESA(0): Supported detailed timing: [ 2761.080] (II) VESA(0): clock: 108.0 MHz Image Size: 338 x 270 mm [ 2761.080] (II) VESA(0): h_active: 1280 h_sync: 1328 h_sync_end 1440 h_blank_end 1688 h_border: 0 [ 2761.080] (II) VESA(0): v_active: 1024 v_sync: 1025 v_sync_end 1028 v_blanking: 1066 v_border: 0 [ 2761.080] (II) VESA(0): Ranges: V min: 56 V max: 75 Hz, H min: 30 H max: 81 kHz, PixClock max 145 MHz [ 2761.080] (II) VESA(0): Monitor name: SyncMaster [ 2761.080] (II) VESA(0): Serial No: H9NY608273 [ 2761.080] (II) VESA(0): EDID (in hex): [ 2761.080] (II) VESA(0): 00ffffffffffff004c2d920137314942
It does not crash at all sites and pages, though several more got reported Monday. It has never crashed for my email client. This evening I read a PDF newsletter I received without any issue. It contained images as well as text. It seems quite content dependent.
More crashes... Report ID Date Submitted bp-c1efacd3-11c7-42b8-90ae-b56f30180614 6/13/18 10:16 PM bp-46f060d5-932c-4fdc-bf3d-06c6e0180614 6/13/18 10:09 PM bp-d65d5bb4-941d-4d6f-abde-5972f1180614 6/13/18 10:07 PM bp-96371d20-666e-4e79-be97-ba92c0180614 6/13/18 10:07 PM bp-0951896b-127e-4c9f-9673-c814e1180612 6/11/18 10:21 PM bp-1e8cf0c2-bf44-4adb-8076-179ba0180612 6/11/18 10:20 PM bp-255b37a1-b2c0-473b-81d6-b97a61180612 6/11/18 10:11 PM bp-a2c41660-3688-4f4b-b381-b01fa1180612 6/11/18 10:11 PM bp-29f53c66-cdd4-41dc-9f2b-0ba791180612 6/11/18 10:09 PM bp-5b77f88e-8650-4039-aa0c-0b95b0180612 6/11/18 10:09 PM bp-f6e3815d-ca61-4ef6-8ae3-3e7ca1180612 6/11/18 10:08 PM bp-a155ba05-8565-435a-933f-f85e11180612 6/11/18 10:08 PM bp-149c71c0-2cc0-424c-ad2c-c50bd1180612 6/11/18 10:07 PM bp-d0875460-8a78-4cc7-b737-9292c1180611 6/11/18 1:52 PM
Modified by cor-el
I've tried it with the VESA driver & Nouveau driver in XOrg, with hardware acceleration and webgl enabled and disabled, and still it crashes!
As a Windows person, I can't personally help with your crash reports.
Do you want to try reinstalling the immediately previous release to see whether that one still works or now has the same issue?
Thank you for responding. I was beginning to wonder if this forum offered actual fixes, or just collected problems for future updates. I could try compiling 57. Does that tell us anything diagnostic and usable? (I just got 60.0.2esr to see if it would build, but it demands more recent versions of too many --with-system-xyz libraries that I have. 55.0.3 was recommended in the BLFS book, but I feel I really want an ESR to cover the Spectre flaws.)
I suppose I could also try 52.8.2 without any of my system libraries, if that would help diagnose it. It would make it bigger and slower.
I did find this: https://support.mozilla.org/en-US/questions/1204598 And tried it, but in my case it still crashed, if perhaps not as often, I can't be sure. It seems pretty reliable that some sites crash it, others don't seem to.
OK, even if you're a Windows guy and even given my Linux system differences, can you tell what firefox is doing when it crashes? Perhaps I can relate that to my Linux system.
Hi paulrogers, that's my problem, when I look at the crash report online, the crashing thread was calling files whose names I don't recognize.
There might be a better forum for people who build Firefox themselves; I've never seen that mentioned here before.
Maybe you don't recognize them, but probably I would, since I built my whole system! My package management can connect files with packages. (I disavow being a Linux expert though!) At least you're looking at and able to get information from the crash reports. 8-) What do you see?
I rebuilt on 6/15 without any "contaminating" system libraries, and it still crashed on some sites, twice in a row at one site, two sites.
One thing I can't get from the reports is the URL that caused the crashes so I can go back and confirm--I'm too old, 74, and my short-term memory is shot!
The links to view the crash reports were in your earlier post (the forum linkifies the crash report IDs): https://support.mozilla.org/questions/1221410#answer-1122497
The URL of the page on which Firefox crashed is not shown in the online crash reports. The developers have access to it, but not web users (for privacy reasons).
OK, I suppose I should be able to get them from the crash reports on my system?
So, what can to tell about where my builds are crashing? Are they always the same (few) places? What was it trying to do at the time? What are the names of those Linux files you don't recogize, that I may be able to identify?
p.s. Thank you for your continued effort.
I'm sorry, I really do not have any idea about your crash reports. Have you looked at the crashing thread section of the first three linked reports? There's nothing for me to go on there.
The Socorro server doesn't have symbols for your Firefox version, so it is impossible to make sense of it. This doesn't look like the official release when I search the buildID.
I would suggest to (re)install the official Firefox 52.8.1 version. Firefox 52.8.1 ESR:
64-bit Firefox for Linux:
- Version=52.8.1
BuildID=20180605153619 - https://mozilla-services.github.io/buildhub/?q=20180605153619
Cor-el, you are correct, this is a handbuilt LFS, Linux from Scratch, system, including Firefox using system libraries where possible, the Linux version of DLLs, to keep a large program a bit smaller and more efficient in systems of modest horsepower. (My most recent tests of Friday morning compiled without any of the system libraries, still crashed multiple times at multiple sites, suggesting that the system libraries are not causing the crashes. But this was just a test to rule out compatibility issues with the system libraries.) If you need the symbol tables, I believe I could recompile it without stripping them, now the default as I understand. Would that supply the information you'd need? It would inflate the size of the code significantly.
jscher2000 said
As a Windows person, I can't personally help with your crash reports. Do you want to try reinstalling the immediately previous release to see whether that one still works or now has the same issue? http://archive.mozilla.org/pub/firefox/releases/52.7.4esr/
OK, I've done that, without stripping the symbol tables as seemed to be suggested in another reply (which inflated a ~50MB install to ~360MB!). 52.7.4 also crashes at one of the sites that has crashed my other 52's. It is appearing to be site specific.
OK, I dumped all the 52.X.Yesr stuff and went back to the 55.0.3 documented in the BLFS book. It failed during configuration because /usr/local/lib/python2.7/multiprocessing/queues.py was missing. (It had been missing when I was trying to make 52 work, but 52's autotools apparently didn't check for it.) I rebuilt python, confirmed queues.py was made, then compiled 55 and went to a site that reliably crashed 52. It didn't crash.
My suspicion is as an ESR 52 needs a fix to its configuration to check for more missing python modules. I aim to confirm that by going back to make 52.8.1 again. I want its Spectre mitigations!
No, it looked likely, but with 52.8.1esr, and the rebuilt python, that site still crashes firefox. I'm obviously more than ready to work on diagnosis, apply a fix, rebuild and test if someone can supply a cogent fix. But it appears to me 52.8.1esr is broken. For the moment I'll have to use 55.0.3 and take my chances with Spectre.
Did you check crash reports from a build that has symbols on the Socorro server?
No, I don't know how to relate that to my local build. I may build my own Linux systems, but my programming career was on IBM mainframes, never involved Unix, and much predated Linux. I'm a firefox user (since v0.7), not developer.