You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2025-12-10 - 19:06

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack7slot7s.osadl.org (updated Wed Dec 10, 2025 12:43:46)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
7308210476,10sleep10-21swapper/107:07:031
636229985,9sleep30-21swapper/307:05:213
740128764,6sleep20-21swapper/207:08:172
719328168,9sleep00-21swapper/007:05:330
765199290,1cyclictest0-21swapper/010:42:200
765199282,24cyclictest0-21swapper/012:20:120
765199280,2cyclictest0-21swapper/007:50:150
765199280,27cyclictest0-21swapper/011:13:130
765199280,27cyclictest0-21swapper/009:45:020
7665992726,1cyclictest0-21swapper/312:11:093
765199262,23cyclictest485-21diskmemload10:22:410
765199261,24cyclictest485-21diskmemload12:32:050
765199260,1cyclictest0-21swapper/010:45:300
765199250,24cyclictest0-21swapper/011:18:330
765199250,24cyclictest0-21swapper/010:25:140
765199250,24cyclictest0-21swapper/007:25:010
765199250,19cyclictest0-21swapper/009:59:380
765199241,22cyclictest485-21diskmemload10:04:470
765199240,2cyclictest0-21swapper/011:46:250
765199240,23cyclictest0-21swapper/009:40:190
765199240,1cyclictest0-21swapper/010:56:190
765199232,20cyclictest485-21diskmemload10:12:520
765199232,20cyclictest16067-21cstates11:50:170
765199232,18cyclictest485-21diskmemload11:55:270
765199232,17cyclictest0-21swapper/007:35:210
765199231,22cyclictest485-21diskmemload10:09:140
765199230,22cyclictest25041-21uname10:30:200
765199230,22cyclictest0-21swapper/012:39:110
765199230,22cyclictest0-21swapper/009:36:420
766599220,21cyclictest0-21swapper/311:10:123
765999220,21cyclictest29175-21sh10:34:592
765999220,1cyclictest0-21swapper/212:30:112
765199224,17cyclictest485-21diskmemload10:36:340
765199222,19cyclictest0-21swapper/011:20:150
765199220,21cyclictest485-21diskmemload11:02:310
765199220,21cyclictest0-21swapper/012:28:180
765199220,18cyclictest0-21swapper/010:15:150
765399200,2cyclictest0-21swapper/110:40:211
765199208,11cyclictest500-21in:imuxsock09:17:130
765199200,1cyclictest0-21swapper/011:44:210
765199200,1cyclictest0-21swapper/007:30:190
766599192,16cyclictest472-21avahi-daemon10:50:193
765999190,19cyclictest0-21swapper/212:19:512
7653991918,1cyclictest0-21swapper/111:29:371
765199190,1cyclictest0-21swapper/012:00:530
765199190,1cyclictest0-21swapper/008:55:410
765199190,18cyclictest0-21swapper/008:25:130
765199190,14cyclictest0-21swapper/008:30:140
765199190,12cyclictest0-21swapper/012:10:280
765999182,15cyclictest0-21swapper/211:40:122
7659991815,3cyclictest0-21swapper/208:15:292
765399180,4cyclictest0-21swapper/109:22:581
765199182,15cyclictest485-21diskmemload12:09:310
765199181,16cyclictest30069-21/usr/sbin/munin11:35:190
765199181,15cyclictest485-21diskmemload09:28:180
765199180,1cyclictest0-21swapper/012:17:530
765199180,17cyclictest0-21swapper/009:30:130
765199180,16cyclictest0-21swapper/011:05:020
765199180,13cyclictest0-21swapper/011:30:210
765999170,3cyclictest141rcu_preempt11:22:552
765199170,1cyclictest0-21swapper/008:00:130
765199170,16cyclictest0-21swapper/010:50:140
766599169,4cyclictest7744-21snmpd10:01:123
7665991615,1cyclictest0-21swapper/309:41:293
765999160,16cyclictest0-21swapper/210:15:532
765199161,14cyclictest0-21swapper/007:40:230
765199160,9cyclictest0-21swapper/009:24:160
765199160,15cyclictest23706-21cut08:50:140
765199160,15cyclictest0-21swapper/008:40:160
7665991510,3cyclictest6669-21turbostat08:15:003
7659991515,0cyclictest0-21swapper/211:26:502
7659991510,5cyclictest37850irq/131-snd_hda12:25:262
7659991510,2cyclictest29545-21cut10:05:262
765999150,15cyclictest0-21swapper/211:53:052
765999150,15cyclictest0-21swapper/209:27:262
7653991510,3cyclictest15099-21ssh10:20:281
765199158,6cyclictest31082-21tr09:05:140
765199152,11cyclictest22069-21sed08:45:200
765199152,10cyclictest4543-21basename08:10:110
765199151,13cyclictest14585-21wc09:50:270
765199150,1cyclictest0-21swapper/009:00:150
765199150,1cyclictest0-21swapper/008:35:140
765199150,1cyclictest0-21swapper/007:15:140
766599144,4cyclictest27147-21systemctl12:00:013
7665991410,3cyclictest14656-21cat09:50:283
7665991410,3cyclictest10167-21cat12:39:593
7665991410,2cyclictest27771-21grep08:55:473
766599140,0cyclictest0-21swapper/309:15:153
765999149,4cyclictest4709-21tr12:35:182
765999148,5cyclictest16885-21grep07:25:272
765999142,9cyclictest7744-21snmpd10:45:162
7659991413,1cyclictest32-21ksoftirqd/212:20:132
7659991411,2cyclictest3754-21/usr/sbin/munin10:10:272
7659991411,1cyclictest0-21swapper/211:13:042
7659991410,3cyclictest1606-21grep09:40:192
765999140,0cyclictest0-21swapper/208:05:132
7653991414,0cyclictest0-21swapper/111:05:141
7653991411,2cyclictest25-21ksoftirqd/111:48:061
7653991410,3cyclictest6496-21date09:45:011
7653991410,3cyclictest31310-21grep10:05:401
7653991410,2cyclictest14028-21grep12:15:291
765199148,5cyclictest17530-21sort07:30:000
765199147,6cyclictest19174-21ssh11:25:210
765199141,12cyclictest485-21diskmemload09:13:370
765199141,11cyclictest7920-21aten2_r7power_e07:10:160
765199140,13cyclictest0-21swapper/007:55:180
766599139,3cyclictest25162-21ls08:50:253
7665991310,2cyclictest3631-21grep10:40:243
7665991310,2cyclictest29667-21ls09:00:233
7665991310,2cyclictest2444-21grep08:05:153
7665991310,2cyclictest24315-21basename11:30:153
7665991310,2cyclictest23745-21sh11:30:003
7665991310,2cyclictest21742-21sed07:35:273
7665991310,2cyclictest14952-21basename11:50:013
7665991310,2cyclictest11851-21cron11:20:013
7665991310,2cyclictest1013-21tr09:10:153
766599130,12cyclictest31301-21grep12:30:173
766599130,12cyclictest25139-21ssh10:30:203
7659991310,2cyclictest9607-21tr08:20:152
7659991310,2cyclictest18312-21cat09:55:002
7659991310,2cyclictest11899-21grep07:15:282
765999130,1cyclictest0-21swapper/211:59:042
765399139,3cyclictest9664-21grep10:15:291
765399139,3cyclictest7173-21grep08:15:181
765399139,3cyclictest2435-21cut08:05:151
765399139,3cyclictest22552-21tr09:30:141
765399139,3cyclictest12349-21cron07:20:001
765399139,3cyclictest10318-21grep10:45:281
7653991310,2cyclictest7897-21sed09:45:181
7653991310,2cyclictest30562-21tail07:55:181
7653991310,2cyclictest22571-21basename07:40:151
7653991310,2cyclictest19159-21grep09:55:181
765399130,12cyclictest13778-21grep09:50:181
766599129,2cyclictest30764-21uname07:55:203
766599129,2cyclictest29582-21sleep07:55:003
766599129,2cyclictest25163-21sed12:25:183
766599129,2cyclictest17250-21cat11:50:283
766599129,2cyclictest17086-21grep09:25:163
766599129,2cyclictest15275-21sed10:20:283
766599129,2cyclictest12435-21grep11:20:153
766599129,2cyclictest11854-21grep07:15:283
766599129,2cyclictest11790-21grep08:25:153
766599123,5cyclictest224-21systemd-journal09:10:003
766599122,9cyclictest30963-21sensors10:05:373
766599120,11cyclictest17201-21grep08:35:173
765999129,2cyclictest32513-21tr08:00:212
765999129,2cyclictest2634-21cat09:10:282
765999129,2cyclictest15715-21grep08:30:272
765999128,3cyclictest27889-21tr12:00:172
765999127,2cyclictest22319-21rm09:57:552
765999125,7cyclictest37850irq/131-snd_hda07:48:542
765999124,5cyclictest7042-21/usr/sbin/munin09:45:262
765999124,3cyclictest224-21systemd-journal09:20:002
7659991211,1cyclictest0-21swapper/212:10:562
7659991210,1cyclictest29095-21ls09:35:242
765999120,12cyclictest0-21swapper/210:27:572
765399129,2cyclictest578-21head08:00:241
765399129,2cyclictest5648-21grep08:10:211
765399129,2cyclictest4763-21ssh09:13:481
765399129,2cyclictest28453-21chrt09:00:151
765399129,2cyclictest28362-21grep12:00:201
765399129,2cyclictest27686-21tr07:50:191
765399129,2cyclictest25936-21sed11:30:271
765399129,2cyclictest24580-21cut12:25:151
765399129,2cyclictest22361-21wc07:40:011
765399129,2cyclictest17814-21ssh10:55:131
765399129,2cyclictest15851-21cut07:25:171
765399129,2cyclictest12928-21tr07:20:181
765399129,2cyclictest1272-21ssh11:10:161
765399129,2cyclictest101262sleep109:19:011
765399128,3cyclictest29545-21grep09:35:281
765399128,3cyclictest11862-21sh11:20:011
765399128,2cyclictest8460-21sort07:10:171
765399122,6cyclictest0-21swapper/111:40:151
7653991210,1cyclictest21583-21tr08:45:181
765199126,5cyclictest3106-21grep08:05:200
765199123,8cyclictest25848-21cat07:45:210
765199122,9cyclictest0-21swapper/008:21:320
189392121,2sleep318979-21apache_processe10:25:123
766599119,1cyclictest4451-21tail11:40:183
766599119,1cyclictest28104-21/usr/sbin/munin09:35:183
766599118,2cyclictest9618-21cat10:15:293
766599118,2cyclictest9480-21grep08:20:143
766599118,2cyclictest7697-21head08:15:203
766599118,2cyclictest32509-21cut08:00:213
766599118,2cyclictest31409-21sed10:35:383
766599118,2cyclictest30469-21ssh12:01:483
766599118,2cyclictest2923-21seq11:39:413
766599118,2cyclictest24034-21cat07:40:273
766599118,2cyclictest23009-21cut09:30:163
766599118,2cyclictest21577-21tr08:45:173
766599118,2cyclictest19397-21cut10:55:253
766599118,2cyclictest18559-21sh09:55:113
766599118,2cyclictest18360-21tail07:30:213
766599118,2cyclictest12986-21grep12:15:193
766599118,2cyclictest12646-21chrt07:20:153
766599118,2cyclictest10884-21chrt10:46:073
766599118,2cyclictest1073-21expr12:05:153
766599113,4cyclictest224-21systemd-journal12:24:443
765999119,1cyclictest5814-21wc08:10:232
765999119,1cyclictest311rcuc/208:39:592
765999118,2cyclictest9496-21tail07:10:272
765999118,2cyclictest7277-21tail11:15:192
765999118,2cyclictest3991-21sh12:07:162
765999118,2cyclictest31927-21grep09:05:192
765999118,2cyclictest31540-21ssh11:08:402
765999118,2cyclictest30577-21grep07:55:182
765999118,2cyclictest29173-21head09:00:182
765999118,2cyclictest27487-21sh11:04:572
765999118,2cyclictest26330-21tr08:55:362
765999118,2cyclictest25515-21taskset08:51:422
765999118,2cyclictest23176-21grep07:40:182
765999118,2cyclictest22560-21cut09:30:142
765999118,2cyclictest21203-21ssh10:58:122
765999118,2cyclictest20233-21rt-features08:40:252
765999118,2cyclictest17611-21chrt07:30:152
765999118,2cyclictest14503-21ssh09:22:492
765999118,2cyclictest1418-21ssh10:38:442
765999118,2cyclictest13543-21taskset11:48:052
765999118,2cyclictest1091-21ssh11:37:272
765999110,10cyclictest25123-21rm10:00:182
765999110,10cyclictest15510-21sh10:20:432
765399118,2cyclictest5795-21ssh12:09:301
765399118,2cyclictest31388-21grep12:30:171
765399118,2cyclictest29385-21cat10:05:001
765399118,2cyclictest22598-21cut11:00:141
765399118,2cyclictest21602-21sh11:55:141
765399118,2cyclictest19794-21ssh09:27:321
765399118,2cyclictest19460-21ssh12:20:191
765399118,2cyclictest17837-21ssh11:50:571
765399118,2cyclictest1422-21sh11:37:501
765399118,2cyclictest14080-21sed11:20:271
765399118,2cyclictest12466-21grep08:25:171
765399117,1cyclictest14379-21idleruntime10:50:201
765399110,9cyclictest0-21swapper/112:38:101
765399110,1cyclictest0-21swapper/112:14:081
765199110,1cyclictest0-21swapper/008:15:160
766599108,1cyclictest6797-21cat09:45:003
766599108,1cyclictest27122sleep310:10:203
766599108,1cyclictest25196-21expr07:45:173
766599108,1cyclictest22511-21rm11:00:123
766599108,1cyclictest20011-21cat08:40:233
766599108,1cyclictest17303-21taskset07:29:433
766599107,2cyclictest27946-21apache_volume11:05:123
766599107,2cyclictest11174-21sh09:20:113
766599107,2cyclictest10132-21cut07:15:003
765999108,1cyclictest24854-21fschecks_time11:30:172
765999108,1cyclictest21672-21cat08:45:182
765999108,1cyclictest13005-21expr07:20:182
765999108,1cyclictest11673-21sshd08:25:022
765999107,2cyclictest13907-21cat10:50:172
765999100,9cyclictest27785-21expr07:50:192
765999100,9cyclictest209732sleep207:35:212
765999100,8cyclictest28671-21kworker/2:010:40:152
765399108,1cyclictest791-21ssh10:37:561
765399108,1cyclictest31299-21cat09:05:151
765399108,1cyclictest25716-21sed10:30:271
765399108,1cyclictest21366-21ssh10:26:421
765399108,1cyclictest10160-21basename08:20:181
765399107,2cyclictest4772-21ssh10:11:041
765399107,2cyclictest26717-21grep08:55:371
765399107,2cyclictest25972-21ntp_states07:45:221
765399107,2cyclictest24908-21head08:50:221
765399107,2cyclictest18666-21tr08:39:591
765399107,2cyclictest17712-21grep07:30:171
765399107,2cyclictest16123-21chrt08:34:061
76659997,1cyclictest14300-21cat08:30:153
76539994,2cyclictest224-21systemd-journal08:43:511
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional