You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-09-04 - 17:14

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack7slot7s.osadl.org (updated Fri Sep 04, 2026 12:43:51)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1510728775,8sleep00-21swapper/007:05:570
1517128661,9sleep20-21swapper/207:06:472
1532728370,9sleep30-21swapper/307:08:493
1510827761,12sleep10-21swapper/107:05:581
58322550,0sleep10-21swapper/111:04:081
1550899300,29cyclictest9845-21cat09:30:001
1551499281,26cyclictest0-21swapper/309:49:353
1550899281,25cyclictest0-21swapper/109:35:251
1551199240,24cyclictest0-21swapper/212:08:322
1550899241,2cyclictest225-21systemd-journal07:50:001
15511992322,1cyclictest0-21swapper/209:48:122
1551199230,23cyclictest0-21swapper/211:22:022
1551199230,23cyclictest0-21swapper/211:22:012
1551499223,3cyclictest22301-21grep12:30:243
1551199220,22cyclictest0-21swapper/210:28:282
15508992220,1cyclictest0-21swapper/109:59:501
15511992120,1cyclictest0-21swapper/210:14:052
1551199211,3cyclictest28206-21grep10:00:152
1551199211,3cyclictest28206-21grep10:00:142
1551499197,2cyclictest8312-21diskmemload09:10:143
15514991917,2cyclictest0-21swapper/309:25:543
15511991917,1cyclictest0-21swapper/211:36:352
1551199190,18cyclictest0-21swapper/211:42:192
1551199190,18cyclictest0-21swapper/209:29:442
1550899190,2cyclictest141rcu_preempt12:01:581
1550899190,2cyclictest0-21swapper/111:23:081
1550899190,2cyclictest0-21swapper/111:23:071
1551499180,17cyclictest0-21swapper/309:40:493
15511991818,0cyclictest0-21swapper/210:55:002
1551199180,18cyclictest0-21swapper/211:27:182
15504991817,1cyclictest0-21swapper/012:13:130
15504991812,5cyclictest18565-21apache210:44:160
1551499179,2cyclictest1149-21sh08:55:013
1551499174,5cyclictest7341-21apt08:00:003
1551499171,6cyclictest225-21systemd-journal07:20:003
15514991710,2cyclictest9330-21head11:25:153
36982160,0sleep20-21swapper/212:39:342
15514991612,3cyclictest0-21swapper/309:53:513
15514991611,1cyclictest0-21swapper/310:15:233
1551199160,16cyclictest0-21swapper/211:04:292
1551199160,15cyclictest0-21swapper/212:01:252
1551199160,0cyclictest0-21swapper/211:51:012
15508991613,2cyclictest0-21swapper/110:30:001
1550899160,15cyclictest0-21swapper/110:05:121
1551499159,2cyclictest28258-21grep07:35:113
1551499159,1cyclictest7752-21grep08:00:133
1551499158,2cyclictest22231-21ssh09:36:423
1551499158,2cyclictest13417-21ssh12:25:233
1551499158,2cyclictest12674-21cat08:10:153
1551499158,2cyclictest11013-21grep12:05:143
1551499158,1cyclictest26081-21sed07:30:143
15514991513,0cyclictest0-21swapper/310:08:293
15514991510,1cyclictest39-21ksoftirqd/310:52:003
15511991514,1cyclictest0-21swapper/211:31:362
1551199150,1cyclictest0-21swapper/210:32:192
1551199150,15cyclictest0-21swapper/209:34:322
1550899150,14cyclictest8312-21diskmemload11:51:181
1550499153,6cyclictest8312-21diskmemload11:05:000
1550499151,4cyclictest225-21systemd-journal10:25:010
1550499150,1cyclictest0-21swapper/012:34:230
1551499148,2cyclictest5147-21ssh12:01:033
1551499148,2cyclictest32407-21taskset07:42:203
1551499148,2cyclictest31583-21ssh11:38:473
1551499148,2cyclictest29201-21cat10:20:113
1551499148,2cyclictest25329-21ssh09:58:243
1551499145,5cyclictest31516-21apt-get11:00:123
1551499145,2cyclictest225-21systemd-journal10:29:333
15511991411,3cyclictest0-21swapper/210:57:552
15511991411,2cyclictest32-21ksoftirqd/207:45:192
15511991411,2cyclictest0-21swapper/212:15:432
15511991410,3cyclictest29792-21tail07:35:242
15511991410,3cyclictest19798-21grep07:15:252
15511991410,2cyclictest27168-21cut11:55:182
1551199140,14cyclictest0-21swapper/211:45:492
1551199140,14cyclictest0-21swapper/210:22:062
1551199140,13cyclictest0-21swapper/210:18:402
1550899141,12cyclictest17443-21ssh09:53:471
1550899140,1cyclictest0-21swapper/110:52:321
1550899140,14cyclictest0-21swapper/112:18:381
1550899140,14cyclictest0-21swapper/111:39:201
1550899140,14cyclictest0-21swapper/109:10:291
1550499147,6cyclictest23690-21munin-run11:14:590
1550499144,3cyclictest16906-21perf09:14:590
15514991310,2cyclictest7233-21grep10:45:173
15514991310,2cyclictest29711-21cut08:45:163
15514991310,2cyclictest21185-21grep08:25:263
15514991310,2cyclictest17685-21wc11:10:313
1551199139,3cyclictest5491-21ssh10:05:232
1551199139,2cyclictest26436-21awk08:40:012
15511991311,1cyclictest13005-21timerwakeupswit09:50:282
15511991310,2cyclictest6443-21grep09:05:142
15511991310,2cyclictest32161-21wc10:40:252
15511991310,2cyclictest28909-21grep09:40:252
15511991310,2cyclictest23004-21wc08:30:182
15511991310,2cyclictest20683-21grep12:30:132
15511991310,2cyclictest18766-21sh09:15:192
15511991310,2cyclictest11604-21grep08:05:262
1551199130,13cyclictest0-21swapper/210:36:022
1550899139,3cyclictest9716-21tr11:45:131
1550899139,3cyclictest2734-21wc11:40:221
1550899139,3cyclictest15628-21grep10:30:241
1550899138,4cyclictest24612-21sort07:25:251
1550899138,4cyclictest24527-21ssh10:35:451
15508991313,0cyclictest0-21swapper/112:30:421
15508991312,1cyclictest0-21swapper/110:01:051
15508991312,1cyclictest0-21swapper/110:01:041
1550899131,11cyclictest0-21swapper/111:58:011
15508991310,2cyclictest7754-21tr08:00:141
15508991310,2cyclictest3603-21head09:45:161
15508991310,2cyclictest25747-21sort08:35:271
15508991310,2cyclictest2307-21sh09:45:001
15508991310,2cyclictest13795-21grep12:25:271
1550899130,1cyclictest0-21swapper/110:13:001
1550899130,1cyclictest0-21swapper/109:20:251
1550899130,13cyclictest0-21swapper/111:14:421
1550899130,12cyclictest0-21swapper/112:21:581
1550499139,3cyclictest20426-21head09:55:160
1550499138,4cyclictest20766-21sh11:31:520
1550499138,2cyclictest8312-21diskmemload09:19:340
1550499133,7cyclictest487-21avahi-daemon12:05:490
1550499132,6cyclictest19070-21awk09:55:000
15504991310,2cyclictest4485-21sed07:50:250
15504991310,2cyclictest25551-21grep08:35:230
15504991310,2cyclictest25281-21latency_hist07:30:000
15504991310,2cyclictest19263-21grep09:35:150
326042120,0sleep30-21swapper/309:24:073
1551499129,3cyclictest270122sleep310:56:533
1551499129,2cyclictest9790-21grep11:45:133
1551499129,2cyclictest722-21tr07:45:143
1551499129,2cyclictest31328-21tail10:40:173
1551499129,2cyclictest2090-21tail11:40:183
1551499129,2cyclictest18235-21tail09:15:153
1551499129,2cyclictest17679-21ssh11:30:213
1551499129,2cyclictest10279-21tr08:05:143
1551499128,4cyclictest537-21snmpd11:18:203
1551499128,3cyclictest26714-21tr11:55:163
1551499128,3cyclictest22970-21sed10:35:183
1551499128,2cyclictest5065-21ssh11:22:423
1551499128,2cyclictest5065-21ssh11:22:423
1551499126,2cyclictest225-21systemd-journal08:34:433
1551499123,5cyclictest225-21systemd-journal08:59:593
1551499120,1cyclictest0-21swapper/311:07:093
1551199129,2cyclictest4954-21apt07:54:592
1551199129,2cyclictest26625-21wc09:40:002
1551199129,2cyclictest24953-21grep11:15:142
1551199129,2cyclictest24790-21sed07:25:252
1551199129,2cyclictest15294-21cron11:10:002
1551199128,3cyclictest19525-21ssh09:55:112
1551199128,3cyclictest13051-21grep12:25:202
1551199128,2cyclictest29013-21cat08:45:012
1551199124,4cyclictest225-21systemd-journal09:05:002
1551199120,1cyclictest0-21swapper/212:10:092
1550899129,2cyclictest3531-21ls09:00:001
1550899129,2cyclictest27123-21chrt10:18:561
1550899129,2cyclictest26965-21tr08:40:141
1550899129,2cyclictest18865-21processes09:15:201
1550899129,2cyclictest11244-21wc09:30:161
1550899128,3cyclictest8312-21diskmemload12:08:381
1550899128,3cyclictest30473-21grep10:20:201
1550899128,3cyclictest24616-21taskset11:34:521
1550899128,3cyclictest21669-21grep07:20:191
1550899128,2cyclictest6711-21grep10:45:131
1550899127,4cyclictest32461-21awk10:40:261
1550899127,4cyclictest13285-21ssh11:08:251
1550899125,4cyclictest28789-21apt-get12:35:111
15508991212,0cyclictest0-21swapper/111:25:131
1550499129,2cyclictest5541-21ssh12:20:310
1550499129,2cyclictest4702-21grep10:05:150
1550499129,2cyclictest3820-21wc12:00:230
1550499129,2cyclictest32060-21sh11:20:000
1550499129,2cyclictest3129-21head08:55:250
1550499129,2cyclictest31231-21grep07:40:160
1550499129,2cyclictest311-21grep11:20:160
1550499129,2cyclictest311-21grep11:20:150
1550499129,2cyclictest27130-21tr09:20:200
1550499129,2cyclictest25494-21tr11:35:150
1550499129,2cyclictest25171-21sed10:55:260
1550499129,2cyclictest19408-21grep07:15:210
1550499129,2cyclictest19257-21cat11:50:230
1550499129,2cyclictest17895-21grep08:20:160
1550499129,2cyclictest12126-21cut12:25:150
1550499128,3cyclictest9808-21wc09:30:000
1550499128,3cyclictest8248-21grep11:05:160
1550499128,2cyclictest23562-21grep08:30:260
1550499120,10cyclictest6223-21grep10:25:170
1550499120,0cyclictest0-21swapper/010:34:510
1551499118,2cyclictest6492-21cat09:05:143
1551499118,2cyclictest5361-21sed12:20:283
1551499118,2cyclictest31517-21rm10:01:413
1551499118,2cyclictest31517-21rm10:01:413
1551499118,2cyclictest31160-21ssh12:35:423
1551499118,2cyclictest29070-21sh12:15:233
1551499118,2cyclictest2849-21chrt07:50:113
1551499118,2cyclictest25005-21uniq08:35:183
1551499118,2cyclictest20839-21expr07:20:143
1551499118,2cyclictest18654-21df11:50:173
1551499118,2cyclictest16206-21sed08:15:233
1551499118,2cyclictest16206-21sed08:15:233
1551499118,2cyclictest15774-21grep07:10:143
1551499118,2cyclictest15752-21ssh10:30:263
1551499118,2cyclictest12753-21ssh10:10:183
1551499118,2cyclictest10916-21tr09:30:153
1551499110,10cyclictest27480-21sed08:40:173
1551499110,10cyclictest19580-21expr12:10:143
1551199118,2cyclictest8860-21sh09:10:052
1551199118,2cyclictest3536-21/usr/sbin/munin08:59:592
1551199118,2cyclictest28020-21ssh09:20:332
1551199118,2cyclictest17536-21ssh11:10:292
1551199118,2cyclictest16860-21sh08:20:002
1551199118,2cyclictest16860-21sh08:20:002
1551199118,2cyclictest15760-21taskset07:10:142
1551199114,3cyclictest225-21systemd-journal08:00:002
1551199112,3cyclictest0-21swapper/208:50:142
1551199110,10cyclictest9208-21ssh12:23:232
1551199110,10cyclictest6142-21chrt10:45:012
1550899119,1cyclictest21216-21grep08:25:261
1550899118,2cyclictest581-21ls08:50:261
1550899118,2cyclictest26136-21sort07:30:151
1550899118,2cyclictest22038-21ssh12:11:081
1550899118,2cyclictest17835-21cron07:15:001
1550899111,9cyclictest25662-21mailstats11:15:191
1550899110,10cyclictest12697-21expr08:10:151
1550499118,2cyclictest9273-21sed08:00:260
1550499118,2cyclictest8639-21ssh10:45:360
1550499118,2cyclictest8415-21ssh09:48:160
1550499118,2cyclictest7289-21grep09:05:190
1550499118,2cyclictest30238-21latency_hist07:40:000
1550499118,2cyclictest28781-21head10:00:170
1550499118,2cyclictest28781-21head10:00:170
1550499118,2cyclictest28186-21cat11:55:260
1550499118,2cyclictest27541-21sort08:40:180
1550499118,2cyclictest27110-21tr09:40:140
1550499118,2cyclictest26390-21if_enp0s31f607:30:170
1550499118,2cyclictest2153-21sed11:40:170
1550499118,2cyclictest20548-21basename07:20:100
1550499118,2cyclictest20485-21ssh10:15:070
1550499118,2cyclictest17848-21idleruntime-cro07:15:000
1550499118,2cyclictest16661-21ssh09:33:560
1550499118,2cyclictest15730-21ssh10:11:360
1550499118,2cyclictest14903-21ssh10:50:170
1550499118,2cyclictest11870-21ssh11:45:380
1550499118,2cyclictest11500-21ssh11:26:130
1550499118,2cyclictest0-21swapper/012:15:300
1550499117,3cyclictest29795-21grep08:45:160
1550499114,5cyclictest8312-21diskmemload12:35:300
1550499112,5cyclictest23489-21ls10:35:210
1551499108,1cyclictest23987-21basename07:25:173
1551499107,2cyclictest18826-21unixbench_multi08:20:253
1551499100,9cyclictest3900-21cut09:00:133
1551199108,1cyclictest32332-21chrt07:41:222
1551199108,1cyclictest18211-21sed08:20:182
1551199107,2cyclictest9600-21chrt08:03:322
1551199107,2cyclictest29601-21sed08:45:142
1551199107,2cyclictest27768-21grep07:35:002
1551199107,2cyclictest19691-21cut08:25:142
1551199107,2cyclictest14474-21munin-plugin-st08:15:002
1550899108,1cyclictest6336-21egrep09:05:141
1550899108,1cyclictest30407-21sendmail_mailqu08:45:211
1550899108,1cyclictest26406-21ssh10:56:231
1550899108,1cyclictest237832sleep108:31:371
1550899108,1cyclictest18247-21cut08:20:181
1550899108,1cyclictest10283-21cstates08:05:151
1550899107,2cyclictest5922-21sh09:05:001
1550899107,2cyclictest5912-21perl07:55:141
1550899107,2cyclictest3049-21cat07:50:131
1550899107,2cyclictest29176-21cat07:35:181
1550899107,2cyclictest15069-21expr08:15:131
1550899107,2cyclictest15069-21expr08:15:121
1550899105,4cyclictest25-21ksoftirqd/107:15:131
1550499108,2cyclictest4927-21nfs4_client09:00:170
1550499108,1cyclictest16591-21chrt08:16:460
1550499108,1cyclictest16591-21chrt08:16:450
1550499108,1cyclictest14131-21chrt08:10:450
1550499108,1cyclictest1285-21wc07:45:170
1550499108,1cyclictest11402sleep008:55:000
1550499107,2cyclictest20198-21grep08:25:160
1550499100,9cyclictest6978-21chrt07:55:380
155119996,2cyclictest21349-21cut07:20:162
155089997,1cyclictest30996-21expr07:40:141
155049997,1cyclictest99502sleep008:05:030
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional