You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-07-14 - 03:50

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot7s.osadl.org (updated Tue Jul 14, 2026 00:43:51)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2890929164,9sleep30-21swapper/319:08:373
2889828976,9sleep20-21swapper/219:08:282
2883928977,8sleep00-21swapper/019:07:420
2878228469,11sleep10-21swapper/119:06:581
162342520,0sleep00-21swapper/022:42:140
171572450,0sleep00-21swapper/000:19:400
2909699270,0cyclictest0-21swapper/023:40:260
2910599260,24cyclictest0-21swapper/222:54:102
2910799230,3cyclictest0-21swapper/320:45:153
29105992222,0cyclictest0-21swapper/221:53:322
2910099223,3cyclictest24122-21/usr/sbin/munin23:45:181
2910099220,2cyclictest0-21swapper/123:35:141
2910099220,15cyclictest0-21swapper/121:30:121
2909699221,18cyclictest0-21swapper/020:05:150
2910599210,21cyclictest0-21swapper/222:34:332
29100992117,3cyclictest16144-21fschecks_count21:25:171
2910099210,19cyclictest0-21swapper/120:40:131
2909699210,4cyclictest0-21swapper/000:20:120
2909699210,20cyclictest0-21swapper/021:46:330
2909699210,16cyclictest0-21swapper/020:25:130
2910599200,20cyclictest0-21swapper/223:56:442
2910599200,20cyclictest0-21swapper/200:07:542
2910599200,19cyclictest0-21swapper/222:17:002
2910099204,1cyclictest141rcu_preempt00:23:121
2910099200,2cyclictest0-21swapper/123:14:251
2909699200,20cyclictest0-21swapper/021:13:210
2909699200,19cyclictest31553-21ssh23:30:120
2910599191,3cyclictest9386-21ssh21:40:202
29105991910,8cyclictest32-21ksoftirqd/221:45:462
2910599190,19cyclictest0-21swapper/222:49:032
29100991919,0cyclictest0-21swapper/100:19:351
2910099190,1cyclictest0-21swapper/122:37:501
2909699195,10cyclictest8867-21munin-node23:51:100
29105991815,2cyclictest0-21swapper/222:24:442
2910599180,18cyclictest0-21swapper/221:12:012
2910099183,14cyclictest22060-21diskmemload23:09:451
2910099180,16cyclictest0-21swapper/122:45:131
2910599171,16cyclictest22060-21diskmemload23:32:492
2910599171,15cyclictest0-21swapper/223:25:142
2910599170,1cyclictest0-21swapper/221:22:592
2910599170,16cyclictest0-21swapper/222:05:112
2910599170,13cyclictest0-21swapper/200:00:432
29100991715,1cyclictest0-21swapper/121:47:101
29100991713,1cyclictest25-21ksoftirqd/122:15:221
2910099170,17cyclictest0-21swapper/121:14:041
2909699171,14cyclictest0-21swapper/021:50:240
29107991615,1cyclictest11975-21kworker/3:023:25:033
2910599160,16cyclictest0-21swapper/222:02:022
2910599160,16cyclictest0-21swapper/221:16:062
2910099161,1cyclictest28729-21apt-get22:50:131
2910099160,16cyclictest0-21swapper/123:04:391
29096991613,2cyclictest904-21munin-run00:10:000
321772150,1sleep125-21ksoftirqd/100:28:191
29107991513,2cyclictest39-21ksoftirqd/322:05:333
2910599154,10cyclictest0-21swapper/200:34:152
2910599150,0cyclictest0-21swapper/223:40:012
2910099150,1cyclictest0-21swapper/122:22:301
2909699159,6cyclictest27579-21apache222:04:130
2909699159,5cyclictest27571-21apache221:33:000
29096991510,3cyclictest7538-21sendmail21:40:000
2909699150,12cyclictest0-21swapper/023:26:560
2910799144,7cyclictest417-21latency_hist21:55:003
2910799143,10cyclictest28075-21apt-get20:15:033
29107991412,1cyclictest0-21swapper/321:19:143
29107991410,3cyclictest18992-21tr19:55:173
29107991410,3cyclictest13874-21grep22:40:263
2910799140,8cyclictest0-21swapper/322:53:253
2910799140,13cyclictest0-21swapper/320:00:143
2910799140,12cyclictest4039-21grep20:30:183
2910799140,12cyclictest4039-21grep20:30:173
29105991413,1cyclictest0-21swapper/222:59:362
29105991411,2cyclictest3001-21cron20:30:012
29105991411,2cyclictest24795-21cat21:30:182
2910599140,6cyclictest0-21swapper/220:10:112
2910599140,14cyclictest0-21swapper/221:59:412
2910099147,4cyclictest25-21ksoftirqd/123:15:111
29100991414,0cyclictest0-21swapper/100:01:551
2910099140,0cyclictest141rcu_preempt00:30:091
2909699147,6cyclictest8797-21munin-run22:00:000
2909699143,4cyclictest225-21systemd-journal19:19:590
29096991413,1cyclictest0-21swapper/023:23:100
29096991410,3cyclictest12538-21basename22:59:590
29096991410,2cyclictest13543-21grep20:50:170
2910799139,3cyclictest8583-21wc23:35:193
2910799139,3cyclictest8233-21grep21:40:143
2910799139,3cyclictest20108-21ssh00:20:253
2910799139,3cyclictest17504-21ssh00:00:163
2910799132,10cyclictest0-21swapper/321:30:133
2910799131,7cyclictest225-21systemd-journal22:33:133
29107991310,2cyclictest9248-21uniq23:55:183
29107991310,2cyclictest880-21cron00:10:013
29107991310,2cyclictest3851-21cut22:35:133
29107991310,2cyclictest23825-21ssh21:10:193
29107991310,2cyclictest16758-21grep23:20:273
29107991310,2cyclictest16169-21grep21:25:173
29107991310,2cyclictest10548-21tr00:15:173
2910599139,3cyclictest19885-21grep22:25:162
2910599133,9cyclictest27579-21apache223:05:542
29105991310,2cyclictest7577-21tr19:30:202
29105991310,2cyclictest25990-21ls00:25:012
29105991310,2cyclictest23365-21cat20:05:002
2910599130,1cyclictest0-21swapper/200:18:312
2910099139,3cyclictest32762-21ssh21:35:171
2910099139,3cyclictest13021-21cat22:40:181
2910099138,3cyclictest25-21ksoftirqd/121:20:101
2910099137,5cyclictest23936-21head23:25:171
2910099137,5cyclictest18998-21tr19:55:171
2910099137,4cyclictest537-21snmpd23:58:141
2910099135,3cyclictest25-21ksoftirqd/121:00:131
29100991310,3cyclictest38150irq/131-snd_hda21:54:291
29100991310,2cyclictest9628-21tr22:00:141
29100991310,2cyclictest1514-21tr00:10:141
29100991310,2cyclictest12723-21grep19:40:201
2910099130,12cyclictest12600-21wc22:59:591
29096991310,2cyclictest32057-21cron23:50:010
29096991310,2cyclictest23247-21cron20:05:010
29096991310,2cyclictest19460-21cut22:05:220
29096991310,2cyclictest13354-21grep23:00:140
29096991310,2cyclictest11356-21grep22:20:170
2909699130,1cyclictest0-21swapper/021:18:030
210022130,1sleep00-21swapper/000:02:060
2910799129,2cyclictest28452-21ssh00:25:273
2910799129,2cyclictest22006-21tr23:05:153
2910799129,2cyclictest2154-21sort00:10:173
2910799129,2cyclictest2050-21grep23:50:263
2910799129,2cyclictest1827-21grep21:55:193
2910799129,2cyclictest18084-21grep19:50:283
2910799129,2cyclictest17717-21grep23:40:263
2910799129,2cyclictest14038-21sh23:20:003
2910799129,2cyclictest11942-21sed22:20:233
2910799128,3cyclictest9165-21grep19:35:143
2910799128,3cyclictest2489-21sh00:30:133
2910799128,3cyclictest20670-21tail21:05:173
2910799128,3cyclictest18796-21cat00:39:593
2910799128,3cyclictest14518-21cut23:00:203
2910799125,5cyclictest28223-21grep22:50:003
29107991210,1cyclictest0-21swapper/323:31:353
2910799120,5cyclictest0-21swapper/322:12:253
2910799120,2cyclictest141rcu_preempt22:00:303
2910599129,2cyclictest7909-21timerandwakeup23:15:252
2910599129,2cyclictest4334-21cat20:30:212
2910599129,2cyclictest4334-21cat20:30:202
2910599129,2cyclictest29764-21rm22:11:362
2910599129,2cyclictest29611-21tr19:10:142
2910599129,2cyclictest15484-21sort23:20:172
2910599129,2cyclictest14114-21cut19:45:152
2910599129,2cyclictest13794-21sed22:40:262
2910599120,12cyclictest0-21swapper/223:35:572
2910099129,2cyclictest8719-21ssh21:40:161
2910099129,2cyclictest29875-21wc20:15:231
2910099129,2cyclictest25-21ksoftirqd/120:35:001
2910099129,2cyclictest25-21ksoftirqd/120:35:001
2910099129,2cyclictest23609-21basename20:05:161
2910099129,2cyclictest20708-21cut21:05:171
2910099129,2cyclictest16392-21grep20:55:221
2910099128,2cyclictest26658-21grep20:10:151
2910099122,0cyclictest141rcu_preempt22:11:401
2910099121,4cyclictest225-21systemd-journal23:25:011
29100991210,1cyclictest7747-21cron20:40:011
2910099120,10cyclictest0-21swapper/121:18:341
2909699129,2cyclictest7320-21rm23:15:190
2909699129,2cyclictest4366-21grep20:30:210
2909699129,2cyclictest4366-21grep20:30:200
2909699129,2cyclictest4104-21grep22:15:280
2909699129,2cyclictest3787-21cron19:25:000
2909699129,2cyclictest32653-21head20:20:270
2909699129,2cyclictest2587-21cut00:30:130
2909699129,2cyclictest23258-21grep23:05:210
2909699129,2cyclictest15414-21tr20:55:140
2909699129,2cyclictest13-21ksoftirqd/023:38:350
2909699128,3cyclictest7973-21grep21:20:190
2909699128,3cyclictest3181-21cat22:35:000
2909699128,3cyclictest17601-21sed19:50:230
2909699128,2cyclictest1724-21cut00:10:160
2909699126,4cyclictest10580-21uptime23:55:280
29096991212,0cyclictest0-21swapper/023:12:560
2909699121,11cyclictest0-21swapper/021:00:170
2910799118,2cyclictest9121-21grep21:20:303
2910799118,2cyclictest6177-21sleep19:29:593
2910799118,2cyclictest5927-21expr20:35:133
2910799118,2cyclictest5536-21grep22:55:193
2910799118,2cyclictest3226-21rm21:36:383
2910799118,2cyclictest31709-21apache_accesses19:15:113
2910799118,2cyclictest2980-21ssh23:12:533
2910799118,2cyclictest25588-21tr23:45:213
2910799118,2cyclictest23809-21grep20:05:173
2910799118,2cyclictest16981-21grep21:45:163
2910799118,2cyclictest14590-21chrt20:51:103
2910799117,3cyclictest27515-21sed20:10:243
2910799117,2cyclictest2966-21grep22:15:173
2910799110,10cyclictest21792-21ssh22:25:563
2910599119,1cyclictest16894-21seq23:01:422
2910599119,1cyclictest16626-21tr19:50:152
2910599119,1cyclictest12346-21grep00:35:222
2910599118,2cyclictest8040-21ls23:55:002
2910599118,2cyclictest7550-21taskset20:37:272
2910599118,2cyclictest5750-21sh22:35:372
2910599118,2cyclictest4714-21grep19:25:162
2910599118,2cyclictest31953-21cat20:20:192
2910599118,2cyclictest30512-21expr23:10:142
2910599118,2cyclictest27734-21ssh23:46:372
2910599118,2cyclictest27334-21expr00:25:182
2910599118,2cyclictest24279-21head20:05:192
2910599118,2cyclictest2412-21tail19:20:182
2910599118,2cyclictest19628-21chrt21:04:352
2910599118,2cyclictest18433-21ssh21:26:152
2910599118,2cyclictest13394-21grep20:50:172
2910599118,2cyclictest12714-21wc20:50:012
2910599118,2cyclictest1268-21ssh00:10:072
2910599118,1cyclictest31921-21ssh21:35:042
2910599117,3cyclictest28776-21tr20:15:162
2910599117,3cyclictest22035-21cron21:10:012
2910099119,1cyclictest25-21ksoftirqd/122:25:281
2910099118,2cyclictest748-21grep23:50:151
2910099118,2cyclictest6162-21cron19:29:591
2910099118,2cyclictest3904-21cat19:25:001
2910099118,2cyclictest3601-21ssh21:55:571
2910099118,2cyclictest31594-21cat19:15:011
2910099118,2cyclictest28612-21ssh22:30:181
2910099118,2cyclictest22433-21ssh22:07:271
2910099118,2cyclictest21722-21ssh23:43:291
2910099118,2cyclictest1581-21grep20:25:181
2910099118,2cyclictest14171-21ssh00:36:281
2910099118,2cyclictest10643-21sed19:35:271
2910099110,7cyclictest0-21swapper/100:05:151
2910099110,11cyclictest0-21swapper/123:30:211
2909699118,2cyclictest9930-21ssh22:38:530
2909699118,2cyclictest31769-21ssh22:13:110
2909699118,2cyclictest28700-21ssh00:25:350
2909699118,2cyclictest28464-21cat20:15:000
2909699118,2cyclictest26082-21ssh22:48:220
2909699118,2cyclictest22858-21ssh22:26:460
2909699118,2cyclictest18145-21ssh21:26:010
2909699118,2cyclictest14793-21sed19:45:180
2909699118,2cyclictest1317-21ssh22:52:480
2909699118,2cyclictest11880-21expr19:40:150
2909699118,2cyclictest11076-21grep00:35:150
2909699117,3cyclictest10728-21grep19:35:270
2909699116,3cyclictest20040-21grep21:05:140
2909699116,3cyclictest15237-21ssh21:44:310
2909699112,3cyclictest18736-21acpi19:55:140
2910799108,1cyclictest8883-21cat20:40:183
2910799108,1cyclictest83112sleep319:32:093
2910799108,1cyclictest3452-21taskset19:20:433
2910799108,1cyclictest18705-21ntpq21:00:213
2910799108,1cyclictest16527-21mailstats20:55:233
2910799108,1cyclictest14627-21tune2fs19:45:163
2910799107,2cyclictest29880-21df19:10:153
2910799100,9cyclictest849-21taskset20:25:033
2910599108,1cyclictest137062sleep219:44:412
2910599108,1cyclictest101012sleep220:44:272
2910599107,2cyclictest32414-21gltestperf19:15:172
2910599107,2cyclictest18757-21apache_processe19:55:142
2910599107,2cyclictest15800-21cut20:55:172
2910099108,1cyclictest7596-21ntpq19:30:211
2910099108,1cyclictest626-21basename19:15:221
2910099108,1cyclictest12924-21cut20:50:151
2910099107,2cyclictest323-21chrt20:21:431
2910099107,2cyclictest11000-21diskstats20:45:171
2909699108,1cyclictest92012sleep020:40:210
2909699108,1cyclictest7832-21python19:30:250
2909699108,1cyclictest4653-21taskset19:25:170
2909699108,1cyclictest10468-21cut20:45:150
2909699107,2cyclictest27503-21sed20:10:240
2909699100,8cyclictest6374-21gltestperf20:35:160
291079997,1cyclictest31251-21cat20:20:143
291079997,1cyclictest11951-21seq19:40:163
291059997,1cyclictest9488-21ls19:35:162
291009997,1cyclictest21936-21expr20:00:171
291009997,1cyclictest17538-21ntpq19:50:221
291009997,1cyclictest14202-21ls19:45:151
290969997,1cyclictest30107-21expr19:10:170
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional